Nonlinear buckling can be evaluated in SAP2000 by running Nonlinear analysis with P-Delta effects and Large Displacements. You can plot displacement of a selected joints vs. the applied loads to see when the structure starts softening and begins to buckle.

Modeling Tips

For symmetrical structures, you may need to introduce some geometric or loading imperfections to initiate buckling.
In the load case (V11 analysis case) definition, request multiple output steps to help convergence and so you can see the buckling.
Using displacement control rather than load control may be necessary if the structure loses load-carrying capacity. This is referring to the control defined in the load case (V11 analysis case), not to how the load is actually applied.
You may need to tighten the Convergence tolerance, say to 1e-6.
Make sure your elements are small enough to capture geometric nonlinear effects. Typically 4 to 8 frame or shell elements per span is needed.
Some buckling cases require converting the nonlinear static analysis case to direct-integration time-history if the instability is severe.

Difference Between Buckling Analysis and Nonlinear Static Analysis to Evaluate Buckling

Linear buckling analysis works by applying perturbations to the undeformed shape, and looking for deflections that could be unstable due to the P-Delta effects under a specified set of loads. The output of the linear buckling analysis is a set of buckling factors and corresponding mode shapes. For each mode shape, the buckling factor is a scale factor that must multiply the applied loads to cause buckling. Similarly, the mode shapes are normalized displacements of the buckled structure and describe how the structure would buckle.

Nonlinear static analysis works by applying the total load in increments and evaluating the stiffness and response at each increment. The stiffness may change from increment to increment due to the following effects:

P-Delta Effects - effect of large tensile or compressive stresses on transverse bending and shear behavior

Large Displacement Effects - deformed shape of the structure is considered when assembling the equilibrium equations

Nonlinear Material Properties - accounts for nonlinear material properties, such as yielding of steel. This behavior is modeled using frame hinges or layered nonlinear shell elements in SAP2000.

Unlike the linear buckling analysis, the nonlinear static analysis provides a "actual response" of the structure (displacements, internal forces, etc.) for each load increment. You can review the deformation path of the structure to determine when the structure starts softening, which may be viewed as initiation of buckling.

We would recommend you to review the following sections of the CSI Analysis Reference Manual that provide good insight into this topic:

chapter Load Cases, section Linear Buckling Analysis

chapter Geometric Nonlinearity, section Overview

chapter Geometric Nonlinearity, section The P-Delta Effect

The linear buckling analysis does not take into account material nonlinearity, so the results may be quite different, depending on how much the material nonlinearity governs the behavior.

Buckling