20. Dumping Problem Formulation and Data

20.1. Why to use the dump tool?

Along with its clients, FORCESPRO provides a tool that allows the user to dump the formulation and actual data of an optimization problem. This information allows to exactly reproduce the same solver for a given formulation and to feed it with exactly the same data to yield exactly the same results (provided it is run on the very same target hardware). The problem formulation and data is stored in “stand-alone” mat or json files, and the problem data can also be saved in binary format. This means there is no need to keep copies of other files that may be used to specify the formulation (such as the dynamic equations), except for formulations relying on external callbacks provided as C code (see External function evaluations in C).

The dump tool may be helpful for a couple of use cases such as:

  • Debugging: a dumped problem allows you to re-run single solver calls without the need to have your full simulation environment up and running.

  • External support: you may send a dumped problem to whomever is in charge of providing support and it will enable that person to exactly reproduce your issue.

  • Testing: keeping dumps of problems that performed as expected can be used to run regression tests to ensure they work as expected after future changes.

Note that, depending on the dump type you choose (see How to use the dump tool?), the dump tool either stores your problem formulation on a symbolic level or keeps a copy of the C code generated by the automatic differentiation tool. Thus, keep the following in mind:

Important

A dumped problem will contain complete information about the solver that you have setup. In particular, it may be used to reverse-engineer your problem formulation (including dynamic model, objective function, constraints etc.). Thus, only share a dumped problem with persons that have a right to obtain this information.

20.2. How to use the dump tool?

The dump tool currently provides three different dump types. Section Symbolic dumps describes the default symbolic dump tool which stores the full symbolic formulation, but requires CasADi v3.5.x to work. Section Legacy dumps describes the so-called legacy dump that is available in the MATLAB client only and does not store the full symbolic formulation. In section Problem dumps from C, you learn how to dump a problem struct (containing the runtime parameters) from C. This is useful when you work on the embedded system and you don’t want to use the MATLAB and Python interface for dumping.

Note

In the Python client, code generation and dumping are not interchangeable, as code generation mutates the CodeOptions and SymbolicModel objects.

To get around this issue, use Python’s copy.deepcopy function to make a copy which will not affected by code generation.

Note, that calling copy.copy makes a shallow copy and will contain references to the original object and will thus also mutate.

20.2.1. Quick reference

This section serves as a quick reference for all client functions related to the dump tool. Please read further below (sections Symbolic dumps and Legacy dumps) to find documentation on how to use the dump tool in context.

20.2.1.1. Matlab

In Matlab, the following functions may be used to dump and load a formulation:

  • High-level interface:

    • ForcesDumpFormulation: dump formulation (symbolic & legacy dumps)

    • ForcesDumpProblem: dump problem data (symbolic & legacy dumps)

    • ForcesDumpAll: dump formulation and problem data (symbolic & legacy dumps)

    • ForcesLoadSymbolicDump: load formulation and problem data (symbolic dumps)

    • ForcesFindDumpedProblems find dumped formulation and problem data (symbolic & legacy dumps)

  • Low-level interface (symbolic dumps only):

    • ForcesDumpFormulationLowLevel: dump formulation

    • ForcesDumpProblemLowLevel: dump problem data

    • ForcesDumpAllLowLevel: dump formulation and problem data

    • ForcesLoadLowLevelDump: load formulation and problem data

    • ForcesFindDumpedProblems: find dumped formulation and problem data

Please refer to the help text of the respective function (by e.g. typing help(ForcesDumpFormulation)) for specific documentation on how to use these functions.

20.2.1.2. Python

In Python, the following functions may be used to dump and load a formulation:

  • forcespro.dump.save_formulation: dump formulation

  • forcespro.dump.save_problem: dump problem data

  • forcespro.dump.save_all: dump formulation and problem data

  • forcespro.dump.load: load formulation and problem data

  • forcespro.dump.find_problems: find dumped formulation and problem data

Please refer to the docstring text of the respective function (by e.g. typing help(forcespro.dump.save_formulation)) for specific documentation on how to use these functions.

20.2.2. Symbolic dumps

Symbolic dumps directly store symbolic expressions of your problem formulation and codeoptions after converting both into the text-based JSON format. From FORCESPRO v6.1.0 onwards, this is the default dump tool if not otherwise specified. Note that this variant reveals your complete problem formulation to anybody with whom you share those JSON files! While you should thus handle those symbolic dumps with care, they offer more flexibility than the legacy dumps and are also available via the Python client of FORCESPRO.

Creating a symbolic dump of a problem consists of two steps:

  1. Dumping the problem formulation: you need to store your model or stages struct, the codeoptions struct and optionally the outputs struct, which can be done even before generating the actual solver code.

  2. Dumping problem data: for each problem instance, the problem params struct needs to be stored. It is possible to store data of multiple problem instances for the same problem formulation (in either a single file or multiple files).

Both steps may also be performed at once.

20.2.2.1. Dumping the problem formulation

For dumping the problem formulation in a symbolic way, just call the following function:

% For the high-level interface:
[tag, fullFilename] = ForcesDumpFormulation( model,codeoptions,outputs,...
                             label,dumpDirectory );

% For the low-level interface:
[tag, fullFilename] = ForcesDumpFormulationLowLevel( stages,codeoptions,outputs,....
                             params,label,dumpDirectory );

In MATLAB, the last argument enables the use of a symbolic dump. This parameter is not required in Python since there is only the symbolic way to dump problems. In Python, the argument modelOrStages refers to the model or stages object, for the high-level interface or the low-level interface, respectively. Pass outputs if your problem formulation contains outputs. Moreover, you may pass an additional label used inside the filenames (or pass an empty string) and dumpDirectory (keyword path in Python) for storing the dumped file (the default is the current working directory). When called this way, the function ForcesDumpFormulation will create a json file in the specified directory containing the passed information. The filename is automatically chosen and will contain the name of your solver, your label, a unique tag, a timestamp as well as the suffix _F, e.g. myFORCESsolver_ABC3DEFGHIJ_20200101120000000_F.json. The returned string fullFilename consists of the dump directory and the filename of the dump.

Note that this function returns a tag that is unique for a given formulation and code options. It is strongly recommended to use it when dumping corresponding problem data. However, the MATLAB and the Python client generate different tags for the same mathematical formulation.

Note that for Python low-level dumps, options and outputs must be set to None (default), as these should be included in the stages object.

Note

For Python low-level dumps, code generation and dumping are not interchangeable, as code generation mutates the CodeOptions object.

To get around this issue, use Python’s copy.deepcopy function to make a copy which will not be affected by code generation.

Note

Also note that formulations making use of CasADi MX expressions (see Automatic differentiation expression class) currently cannot be dumped.

20.2.2.2. Dumping problem data

Assuming your generated FORCESPRO solver is called myFORCESsolver and you are calling it with the following command

[output, exitflag, info] = myFORCESsolver( problem );

then dumping the problem data of any problem instance is as simple as calling

% For the high-level interface:
fullFilename = ForcesDumpProblem( problems,tag,dumpDirectory );

% For the low-level interface:
fullFilename = ForcesDumpProblemLowLevel( problems,tag,dumpDirectory );

You can pass as problems either a single problem parameter struct (dictionary in Python) or a cell array (list in Python) of problem parameter structs. Besides, the unique tag that has been generated when dumping the problem formulation is required. The third argument dumpDirectory for storing the dumped problem data is optional (with the default being the current working directory). The functions ForcesDumpProblem or forcespro.dump.save_formulation will create a json file in the specified directory containing the passed information. The filename is automatically chosen and will contain the name of your solver, the unique tag (including any label passed when dumping the formulation), a timestamp as well as the suffix _P, e.g. myFORCESsolver_ABC3DEFGHIJ_20200101120001000_P.json. The returned string fullFilename consists of the dump directory and the filename of the dump.

There is no limit on the number of problem instances that you may dump that way.

20.2.2.3. Dumping the problem formulation and data at once

For dumping both the problem formulation and all problem data at once in a symbolic way to a single json file, just call the following function:

% For the high-level interface:
[tag, fullFilename] = ForcesDumpAll( model,codeoptions,outputs,...
                     label,dumpDirectory,problems );

% For the low-level interface:
[tag, fullFilename] = ForcesDumpAllLowLevel( stages,codeoptions,outputs,...
                    params,label,dumpDirectory,problems );

The arguments and behavior are the same as for the functions used to dump formulation and problem separately (see Dumping the problem formulation and Dumping problem data). The only difference is that formulation and problems are dumped to a single file with suffix _A, e.g. myFORCESsolver_ABC3DEFGHIJ_20200101120000000_A.json

20.2.2.4. Running a dumped problem

After you have dumped a problem formulation and at least one set of problem data, you can use either a matching pair of _F/_P files or a single _A file in JSON format to exactly reproduce your solver and problem instances. To do so, you need to perform the following two steps:

  1. Load problem formulation and data from JSON file or files by calling:

% For the high-level interface:
[model, codeoptions, outputs, additionalData, problems] = ...
              ForcesLoadSymbolicDump( formulationFilename,problemFilenames );

% For the low-level interface:
[stages, codeoptions, outputs, params, problems] = ForcesLoadLowLevelDump( formulationFilename, problemFilenames );

problemFilenames may either be a single file name or a cell array (list in Python) containing all the problem data set that you want to load. In case you have dumped both formulation and problem data set(s) at once within a single file, just pass that one as formulationFilename and do not specify problemFilenames. The returned problems variable is an array (list in Python) containing all problem sets found in the dumps.

  1. Re-generating and running the solver with dumped information by simply calling:

% For the high-level interface:
FORCES_NLP( model,codeoptions,outputs );

% For the low-level interface:
generateCode( stages,params,codeoptions,outputs );

myFORCESsolver( problems(1) );
% and more problem instances if present

Tip

To get the filenames of dumped problems and formulations in a directory, simply use the function:

[formulationFilename, problemFilename] = ForcesFindDumpedProblems( tag,dumpDirectory );

Both arguments are optional. If tag is not given, the function returns any dumped filenames regardless of their tag. The default dumpDirectory is the current working directory. problemFilenames is a cell array (list in Python) of problem filenames.

Important

MATLAB support for low-level dumps has been added in FORCESPRO version 6.1.0, and low-level dumps created with an earlier version (from Python) can’t be loaded in MATLAB.

20.2.2.5. Limitations of symbolic dumps

Symbolic dumps only work with CasADi v3.5.x for reasons beyond our control, which is why we currently do not plan to extend support to CasADi v2.4.2 or MathWorks’ Symbolic Math Toolbox.

20.2.3. Legacy dumps

Legacy dumps are available for the MATLAB client only and store a pre-processed problem formulation including C code generated by the automatic differentiation tool. This variant is somewhat less explicit and is supposed to work with all supported AD tools.

Creating a legacy dump of a problem consists of two steps:

  1. Dumping the problem formulation: once a new solver has been generated, a formulation struct, the codeoptions struct and optionally the outputs struct need to be stored.

  2. Dumping problem data: for each problem instance, the problem params struct needs to be stored. It is possible to store data of multiple problem instances for the same problem formulation.

20.2.3.1. Dumping the problem formulation

For dumping the problem formulation, the following three steps need to be taken:

  1. Enabling creation of a formulation dump: This is done by using the option

    codeoptions.dump_formulation = 1;

  2. Obtaining the dumped formulation: Calling FORCES_NLP with the before-mentioned code option enabled will make it return a formulation struct as third output argument

    [stages, codeoptions, formulation] = FORCES_NLP( model, codeoptions, outputs );

  3. Storing the necessary structs into a file: After calling FORCES_NLP, you should use the following function to store both the formulation and codeoptions struct

    [tag, fullFilename] = ForcesDumpFormulation( formulation,codeoptions,outputs,label,dumpDirectory,ForcesDumpType.LegacyDumpGeneratedC );

Pass outputs if your problem formulation contains outputs. Moreover, you may pass an additional label used inside the filenames (or pass an empty string) and dumpDirectory for storing the dumped formulation (the default is the current working directory). The function ForcesDumpFormulation will create a mat file in the specified directory containing the passed information. The filename is automatically chosen and will contain the name of your solver, your label, a unique tag, a timestamp as well as the suffix _F, e.g. myFORCESsolver_ABC3DEFGHIJ_20200101120000000_F.mat. The returned fullFilename is a string consisting of the directory and the filename of the dump.

Note that this function returns a tag that is unique for a given formulation and code options. It is strongly recommended to use it when dumping corresponding problem data. Note that the MATLAB and the Python client generate different tags for the same mathematical formulation.

20.2.3.2. Dumping problem data

Assuming your generated FORCESPRO solver is called myFORCESsolver and you are calling it with the following command

[output, exitflag, info] = myFORCESsolver( problem );

then dumping the problem data of any problem instance is as simple as calling

fullFilename = ForcesDumpProblem( problem,tag,dumpDirectory,ForcesDumpType.LegacyDumpGeneratedC );

Here, you need to provide both the problem parameter struct as well as the unique tag that has been generated when dumping the problem formulation. The third argument dumpDirectory for storing the dumped problem data is optional (with the default being the current working directory). The function ForcesDumpProblem will create a mat file in the specified directory containing the passed information. The filename is automatically chosen and will contain the name of your solver, the unique tag (including any label passed when dumping the formulation), a timestamp as well as the suffix _P, e.g. myFORCESsolver_ABC3DEFGHIJ_20200101120001000_P.mat. The returned fullFilename is a string consisting of the directory and the filename of the dump.

There is no limit on the number of problem instances that you may dump that way.

20.2.3.3. Running a dumped problem

After you have dumped a problem formulation and at least one set of problem data, you can use those mat files to exactly reproduce your solver and problem instances. To do so, you need to perform the following two steps (where we assume you have stored the two files myFORCESsolver_ABC3DEFGHIJ_20200101120000000_F.mat and myFORCESsolver_ABC3DEFGHIJ_20200101120001000_P.mat at a location in your MATLAB path):

  1. Re-generate the FORCESPRO solver by loading the formulation mat file and using its content to call the code generation:

    F = load('myFORCESsolver_ABC3DEFGHIJ_20200101120000000_F.mat');
FORCES_NLP( F.formulation,F.codeoptions,F.outputs );

    This will re-create the solver MEX function myFORCESsolver. Note that the third input struct containing the outputs is only available if you included it into your dump.

  2. Running the solver with dumped problem data by loading the data mat file and using its content to call the generated solver:

    P = load('myFORCESsolver_ABC3DEFGHIJ_20200101120001000_P.mat');
myFORCESsolver( P.problem );

    You may repeat this step for as many problem instances as you have dumped.

Tip

To get the filenames of dumped problems and formulations in a directory, simply use the function:

[formulationFilename, problemFilename] = ForcesFindDumpedProblems( tag,dumpDirectory );

Both arguments are optional. If tag is not given, the function returns any dumped filenames regardless of their tag. The default dumpDirectory is the current working directory. problemFilenames is a cell array of problem filenames.

20.2.3.4. Limitations of legacy dumps

Legacy dumps have the following limitations:

  • They are only available via the MATLAB client of FORCESPRO.

  • They cannot be used if you pass external functions in form of C code.

  • They cannot be used in combination with CasADi MX expressions (see Automatic differentiation expression class).

These limitations can be overcome by using a symbolic dump (see Symbolic dumps).

20.2.4. Problem dumps from C

Problem dumps from C consider only the params (or problem) struct containing the runtime parameters that are passed to the solver in each solver call. In order to save or load params (or problem) structs, you can call the dump tool from any C script. This offers the opportunity to work directly on the embedded system. The data is stored in binary format. The problem dumps from C use the msgpack-c library.

Important

The problem dumps from C require dynamic memory allocation. This is because of the dependency on the msgpack-c library. Please check if your embedded platform supports dynamic memory allocation.

Before dumping problems from C for the first time, you need to install msgpack as described in Download and install msgpack for C. Then, the dumping procedure consists of the following three steps:

  1. Enabling the generation of the dump functions: When generating your solver, you need to set a codeoption in order to enable the generation of the dump functions.

  2. Dumping problem data: Call the generated serialize function in order to dump a params struct. Exactly one params struct can be stored at a time.

  3. Loading problem data: Call the generated deserialize function in order to load a params struct. Exactly one params struct can be stored at a time.

20.2.4.1. Download and install msgpack for C

Since the dump tool for problems from C requires msgpack, make sure you installed the library. You can either clone the github repository or you can simply download msgpack as a zip file. For the installation of the library, you need gcc >= 4.1.0 and cmake >= 2.8.1.

How to install:

  • Windows:

    1. Run cmake . in your terminal from the msgpack-c-c_master folder.

    2. Open the generated msgpack.sln file (located in the same folder) in Visual Studio and click Build. The generated library is located in msgpack-c-c_master/Debug.

  • Other platforms:

    1. Run cmake . in your terminal from the msgpack-c-c_master folder.

    2. Run make. The generated library is located in msgpack-c-c_master.

For more information about the installation of msgpack, see the README.md on github.

20.2.4.2. Enable the Generation of the Dump Functions

Before generating your solver you need to enable the generation of the dump functions. You can specify whether you want to dump from your host platform or/and from your target platform. For dumping from the host platform set:

codeoptions.serializeCParamsHost = 1;

and for dumping from your specified target platform set:

codeoptions.serializeCParamsTarget = 1;

20.2.4.3. Dumping Problem Data

This section explains how to write and run a C script that dumps your problem data based on the high-level basic example (see High-level interface: Basic example). We assume your C script is in the same folder as your generated solver.

You can download the code of this example script to try it out for yourself by clicking here.

  1. Include your solver header called <solvername>.h. In the solver header, the <solvername>_params struct and the dumping routines <solvername>_serialize and <solvername>_deserialize are defined. Note that the params struct for binary problems is called <solvername>_binaryparams and for mixed integer problems it is <solvername>_integerparams.

#include "FORCESNLPsolver/include/FORCESNLPsolver.h"

  1. Create a params struct and fill it with your data:

/* create params struct */
FORCESNLPsolver_params params;

/* fill params struct with data */
params.xinit[0] = -4.;
params.xinit[1] = 2.;
for (int i = 0; i < 33; i++)
{
    params.x0[i] = 0.0;
}

  1. Choose a filename for the dump and call the serialization routine:

const char filename[] = "dump.msgpack";
int returnvalueSerialize = FORCESNLPsolver_serialize(&params, filename);

  1. Compile your script:

$ <Compiler_exec> my_C_dump_script.c <compiled_solver> -L<msgpack_lib_path> -l<msgpack_lib> <additional_libs>

Where:

  • <Compiler_exec> is your compiler (for example gcc)

  • my_C_dump_script.c is your script that calls the serialize function (for example serializationCParams_HighLevel_BasicExample.c)

  • <compiled_solver> is your compiled solver library (static or shared):

    • For Linux/MacOS/MinGW it is libFORCESNLPsolver.a or libFORCESNLPsolver.so in the lib or lib_target directory

    • For Windows it is FORCESNLPsolver_static.lib or FORCESNLPsolver.lib in the lib or lib_target directory

  • <msgpack_lib_path> specifies your path to the compiled msgpack library

    • For Linux/MacOS/MinGW it is the path to your msgpack-c-c_master folder

    • For Windows it is the path to your msgpack-c-c_master/Debug folder

  • <msgpack_lib> specifies the name of the compiled msgpack library

    • For the static library on Windows set it to msgpackc_import

    • Otherwise, it is msgpackc

  • <additional_libs> are possible libraries that need to be linked to resolve existing dependencies.

    • For Linux/MacOS it’s usually necessary to link the math library (-lm)

    • For Windows you usually need to link the iphlpapi.lib library (it’s distributed with the Intel Compiler, MinGW as well as MATLAB) and sometimes some additional intel libraries (those are included in the FORCESPRO client under the folder libs_Intel – if missing they are downloaded after code generation)

The following shows how to compile the dump example script on Linux:

$ gcc serializationCParams_HighLevel_BasicExample.c FORCESNLPsolver/lib/libFORCESNLPsolver.so -L/path/to/msgpack-c -lmsgpackc -lm

20.2.4.4. Loading Problem Data

This section explains how to write and run a C script that loads a dumped C params struct based on the high-level basic example (see High-level interface: Basic example). We assume your C script is in the same folder as your generated solver.

You can download the code of this example script to try it out for yourself by clicking here.

  1. Include your solver header called <solvername>.h.

#include "FORCESNLPsolver/include/FORCESNLPsolver.h"

  1. Create an empty params struct:

FORCESNLPsolver_params dumped_params;

  1. Call the deserialization routine and pass the filename you chose when dumping the problem data:

int returnvalueDeserialize = FORCESNLPsolver_deserialize(&dumped_params, filename);

  1. Compile your script:

$ <Compiler_exec> my_C_dump_script.c <compiled_solver> -L<msgpack_lib_path> -l<msgpack_lib> <additional_libs>

Where:

  • <Compiler_exec> is your compiler (for example gcc)

  • my_C_dump_script.c is your script that calls the deserialize function (for example serializationCParams_HighLevel_BasicExample.c)

  • <compiled_solver> is your compiled solver library (static or shared):

    • For Linux/MacOS/MinGW it is libFORCESNLPsolver.a or libFORCESNLPsolver.so in the lib or lib_target directory

    • For Windows it is FORCESNLPsolver_static.lib or FORCESNLPsolver.lib in the lib or lib_target directory

  • <msgpack_lib_path> specifies your path to the compiled msgpack library

    • For Linux/MacOS/MinGW it is the path to your msgpack-c-c_master folder

    • For Windows it is the path to your msgpack-c-c_master/Debug folder

  • <msgpack_lib> specifies the name of the compiled msgpack library

    • For the static library on Windows set it to msgpackc_import

    • Otherwise, it is msgpackc

  • <additional_libs> are possible libraries that need to be linked to resolve existing dependencies.

    • For Linux/MacOS it’s usually necessary to link the math library (-lm)

    • For Windows you usually need to link the iphlpapi.lib library (it’s distributed with the Intel Compiler, MinGW as well as MATLAB) and sometimes some additional intel libraries (those are included in the FORCESPRO client under the folder libs_Intel – if missing they are downloaded after code generation)

The following shows how to compile the dump example script on Linux:

$ gcc serializationCParams_HighLevel_BasicExample.c FORCESNLPsolver/lib/libFORCESNLPsolver.so -L/path/to/msgpack-c -lmsgpackc -lm

20.2.4.5. Dumping and Loading Problem Data from the MATLAB Client

It is possible to dump a params (or problem) struct into the msgpack format, so that it can be loaded from a FORCESPRO solver in C, directly from the MATLAB client. Likewise, a params struct dumped from C can be loaded from the MATLAB client by enabling the following codeoption:

codeoptions.serializeCParamsMex = 1;
% need to set codeoptions.serializeCParamsHost = 1 as well

This enables generation of two additional MEX files <solvername>_serialize_mex.c and <solvername>_deserialize_mex.c within the interface folder of the generated solver. Additionally, the client tries to MEX compile those two files assuming a msgpack-c library is available. If this library cannot be found on your system, that auto-compilation will fail and you need to MEX compile those functions manually.

Note

Those two generated MEX files for (de)serialization are specific to a given FORCESPRO solver and cannot be used to (de)serialize params structs of any other FORCESPRO solver.

Assuming those two MEX functions have been successfully generated and compiled, you can dump a MATLAB params struct into a file in msgpack format named filename as follows:

success = <solvername>_serialize_mex(params, filename);

success will be 1 if the params struct has been successfully serialized, and 0 otherwise (which is different from the returnvalue returned by the corresponding plain C function).

Similarly, a file named filename containing a params struct in msgpack format can be loaded into a MATLAB struct loadedParams by calling

[success, loadedParams] = <solvername>_deserialize_mex(filename);

Note that loadedParams will contain meaningful data only if success is 1 (which is different from the returnvalue returned by the corresponding plain C function)!

20.2.4.6. Dumping and Loading Problem Data from the Python Client

It is possible to dump a problem (or params) dictionary into the msgpack format, so that it can be loaded from a FORCESPRO solver in C, directly from the Python client. This functionality is automatically included into any generated FORCESPRO solver and can be called as follows:

# assumes a solver object as returned by the function generate_solver
solver.serialize_c_params(problem, filename)

This line of code will store a problem dictionary into a file in msgpack format named filename. Rather than returning a success flag, the Python client would throw an exception in case of an error.

Likewise, a problem struct dumped from C can be loaded from the Python client by calling:

# assumes a solver object as returned by the function generate_solver
loaded_problem = solver.deserialize_c_params(filename)

This loads the struct into the dictionary loaded_problem assuming that the file called filename contains a problem struct in msgpack format.

Note

Serialization and deserialization of C parameters from Python requires the msgpack package to be available in your Python installation, otherwise an exception will be thrown.