16. Dumping Problem Formulation and Data

16.1. Why to use the dump tool?

Along with its clients, FORCESPRO provides a tool that allows the user to dump the formulation and actual data of an optimization problem. This information allows to exactly reproduce the same solver for a given formulation and to feed it with exactly the same data to yield exactly the same results (provided it is run on the very same target hardware). The problem formulation and data is stored in “stand-alone” mat or json files, and the problem data can also be saved in binary format. This means there is no need to keep copies of other files that may be used to specify the formulation (such as the dynamic equations), except for formulations relying on external callbacks provided as C code (see External function evaluations in C).

The dump tool may be helpful for a couple of use cases such as:

  • Debugging: a dumped problem allows you to re-run single solver calls without the need to have your full simulation environment up and running.

  • External support: you may send a dumped problem to whomever is in charge of providing support and it will enable that person to exactly reproduce your issue.

  • Testing: keeping dumps of problems that performed as expected can be used to run regression tests to ensure they work as expected after future changes.

Note that, depending on the dump type you choose (see How to use the dump tool?), the dump tool either stores your problem formulation on a symbolic level or keeps a copy of the C code generated by the automatic differentiation tool. Thus, keep the following in mind:

Important

A dumped problem will contain complete information about the solver that you have setup. In particular, it may be used to reverse-engineer your problem formulation (including dynamic model, objective function, constraints etc.). Thus, only share a dumped problem with persons that have a right to obtain this information.

16.2. How to use the dump tool?

The dump tool currently provides three different dump types. Section Legacy dumps describes the so-called legacy dump that is available in the MATLAB client only and does not store the full symbolic formulation. Section Symbolic dumps describes the more recent symbolic dump that is also available in the Python client, stores the full symbolic formulation, but requires CasADi v3.5.1 to work. In section Problem dumps from C, you learn how to dump a problem struct (containing the runtime parameters) from C. This is useful when you work on the embedded system and you don’t want to use the MATLAB and Python interface for dumping.

16.2.1. Legacy dumps

Legacy dumps are available for the MATLAB client only and store a pre-processed problem formulation including C code generated by the automatic differentiation tool. This variant is somewhat less explicit and is supposed to work with all supported AD tools, and is therefore still the default dump type if not specified otherwise by the user.

Creating a legacy dump of a problem consists of two steps:

  1. Dumping the problem formulation: once a new solver has been generated, a formulation struct, the codeoptions struct and optionally the outputs struct need to be stored.

  2. Dumping problem data: for each problem instance, the problem params struct needs to be stored. It is possible to store data of multiple problem instances for the same problem formulation.

16.2.1.1. Dumping the problem formulation

For dumping the problem formulation, the following three steps need to be taken:

  1. Enabling creation of a formulation dump: This is done by using the option

    codeoptions.dump_formulation = 1;
    
  2. Obtaining the dumped formulation: Calling FORCES_NLP with the before-mentioned code option enabled will make it return a formulation struct as third output argument

    [stages, codeoptions, formulation] = FORCES_NLP( model, codeoptions, outputs );
    
  3. Storing the necessary structs into a file: After calling FORCES_NLP, you should use the following function to store both the formulation and codeoptions struct

    [tag, fullFilename] = ForcesDumpFormulation( formulation,codeoptions,outputs,label,dumpDirectory );
    

All but the first two arguments are optional. Pass outputs if your problem formulation contains outputs. Moreover, you may pass an additional label used inside the filenames (or pass an empty string) and dumpDirectory for storing the dumped formulation (the default is the current working directory). The function ForcesDumpFormulation will create a mat file in the specified directory containing the passed information. The filename is automatically chosen and will contain the name of your solver, your label, a unique tag, a timestamp as well as the suffix _F, e.g. myFORCESsolver_ABC3DEFGHIJ_20200101120000000_F.mat. The returned fullFilename is a string consisting of the directory and the filename of the dump.

Note that this function returns a tag that is unique for a given formulation and code options. It is strongly recommended to use it when dumping corresponding problem data.

16.2.1.2. Dumping problem data

Assuming your generated FORCESPRO solver is called myFORCESsolver and you are calling it with the following command

[output, exitflag, info] = myFORCESsolver( problem );

then dumping the problem data of any problem instance is as simple as calling

fullFilename = ForcesDumpProblem( problem,tag,dumpDirectory );

Here, you need to provide both the problem parameter struct as well as the unique tag that has been generated when dumping the problem formulation. The third argument dumpDirectory for storing the dumped problem data is optional (with the default being the current working directory). The function ForcesDumpProblem will create a mat file in the specified directory containing the passed information. The filename is automatically chosen and will contain the name of your solver, the unique tag (including any label passed when dumping the formulation), a timestamp as well as the suffix _P, e.g. myFORCESsolver_ABC3DEFGHIJ_20200101120001000_P.mat. The returned fullFilename is a string consisting of the directory and the filename of the dump.

There is no limit on the number of problem instances that you may dump that way.

16.2.1.3. Running a dumped problem

After you have dumped a problem formulation and at least one set of problem data, you can use those mat files to exactly reproduce your solver and problem instances. To do so, you need to perform the following two steps (where we assume you have stored the two files myFORCESsolver_ABC3DEFGHIJ_20200101120000000_F.mat and myFORCESsolver_ABC3DEFGHIJ_20200101120001000_P.mat at a location in your MATLAB path):

  1. Re-generate the FORCESPRO solver by loading the formulation mat file and using its content to call the code generation:

    F = load('myFORCESsolver_ABC3DEFGHIJ_20200101120000000_F.mat');
    FORCES_NLP( F.formulation,F.codeoptions,F.outputs );
    

    This will re-create the solver MEX function myFORCESsolver. Note that the third input struct containing the outputs is only available if you included it into your dump.

  2. Running the solver with dumped problem data by loading the data mat file and using its content to call the generated solver:

    P = load('myFORCESsolver_ABC3DEFGHIJ_20200101120001000_P.mat');
    myFORCESsolver( P.problem );
    

    You may repeat this step for as many problem instances as you have dumped.

Tip

To get the filenames of dumped problems and formulations in a directory, simply use the function:

[formulationFilename, problemFilename] = ForcesFindDumpedProblems( tag,dumpDirectory );

Both arguments are optional. If tag is not given, the function returns any dumped filenames regardless of their tag. The default dumpDirectory is the current working directory. problemFilenames is a cell array of problem filenames.

16.2.1.4. Limitations of legacy dumps

Legacy dumps have the following limitations:

  • They are only available via the MATLAB client of FORCESPRO.

  • They cannot be used if you pass external functions in form of C code.

These limitations can be overcome by using a symbolic dump.

16.2.2. Symbolic dumps

Symbolic dumps direcly store symbolic expressions of your problem formulation and codeoptions after converting both into the text-based JSON format. This variant thus reveals your complete problem formulation to anybody with whom you share those JSON files! While you should thus handle those symbolic dumps with care, they offer more flexibility than the legacy dumps and are also available via the Python client of FORCESPRO.

Creating a symbolic dump of a problem consists of two steps:

  1. Dumping the problem formulation: you need to store your model struct, the codeoptions struct and optionally the outputs struct, which can be done even before generating the actual solver code.

  2. Dumping problem data: for each problem instance, the problem params struct needs to be stored. It is possible to store data of multiple problem instances for the same problem formulation (in either a single file or multiple files).

Both steps may also be performed at once.

16.2.2.1. Dumping the problem formulation

For dumping the problem formulation in a symbolic way, just call the following function:

[tag, fullFilename] = ForcesDumpFormulation( model,codeoptions,outputs,...
                             label,dumpDirectory,ForcesDumpType.DumpSymbolics );
tag,full_filename = forcespro.dump.save_formulation(model, codeoptions, outputs, \
                                            label, dump_directory)

In MATLAB, the last argument enables the use of a symbolic dump. This parameter is not required in Python since there is only the symbolic way to dump problems so far. Pass outputs if your problem formulation contains outputs. Moreover, you may pass an additional label used inside the filenames (or pass an empty string) and dumpDirectory for storing the dumped file (the default is the current working directory). When calling this way, the function ForcesDumpFormulation will create a json file in the specified directory containing the passed information. The filename is automatically chosen and will contain the name of your solver, your label, a unique tag, a timestamp as well as the suffix _F, e.g. myFORCESsolver_ABC3DEFGHIJ_20200101120000000_F.json. The returned string fullFilename consists of the dump directory and the filename of the dump.

Note that this function returns a tag that is unique for a given formulation and code options. It is strongly recommended to use it when dumping corresponding problem data. However, the MATLAB and the Python client generate different tags for the same mathematical formulation.

16.2.2.2. Dumping problem data

Assuming your generated FORCESPRO solver is called myFORCESsolver and you are calling it with the following command

[output, exitflag, info] = myFORCESsolver( problem );
output, exitflag, info = my_forces_solver.solve(problem);

then dumping the problem data of any problem instance is as simple as calling

fullFilename = ForcesDumpProblem( problems,tag,dumpDirectory,ForcesDumpType.DumpSymbolics );
full_filename = forcespro.dump.save_formulation(problems, tag, dump_directory)

In MATLAB, the last argument enables the use of a symbolic dump. This parameter is not required in Python since there is only the symbolic way to dump problems so far. You need to provide either a single problem parameter struct (dictionary in Python) or an array (list in Python) of problem parameter structs using problems. Besides, the unique tag that has been generated when dumping the problem formulation is required. The third argument dumpDirectory for storing the dumped problem data is optional (with the default being the current working directory). The functions ForcesDumpProblem or forcespro.dump.save_formulation will create a json file in the specified directory containing the passed information. The filename is automatically chosen and will contain the name of your solver, the unique tag (including any label passed when dumping the formulation), a timestamp as well as the suffix _P, e.g. myFORCESsolver_ABC3DEFGHIJ_20200101120001000_P.json. The returned string fullFilename consists of the dump directory and the filename of the dump.

There is no limit on the number of problem instances that you may dump that way.

16.2.2.3. Dumping the problem formulation and data at once

For dumping both the problem formulation and all problem data at once in a symbolic way, just call the following function:

[tag, fullFilename] = ForcesDumpAll( model,codeoptions,outputs,...
                     label,dumpDirectory,problems,ForcesDumpType.DumpSymbolics );
tag, full_filename = forcespro.dump.save_all(model, codeoptions, outputs, \
                      label, dump_directory, problems)

In MATLAB, the last argument enables the use of a symbolic dump. This parameter is not required in Python since there is only the symbolic way to dump problems so far. You need to provide the model and the codeoptions containing your formulation. Pass outputs if your problem formulation contains outputs. Moreover, you may pass an additional label used inside the filenames (or pass an empty string) and dumpDirectory for storing the dumped file (the default is the current working directory). Furthermore, problems may be either a single set of problem data or a cell array (list in Python) of many problem data sets that you want to dump along with the problem formulation.

When calling this way, the function ForcesDumpAll will create a json file in the specified directory containing the passed information. The filename is automatically chosen and will contain the name of your solver, your label, a unique tag, a timestamp as well as the suffix _A, e.g. myFORCESsolver_ABC3DEFGHIJ_20200101120000000_A.json. The returned string fullFilename consists of the dump directory and the filename of the dump.

Note that this function returns a tag that is unique for a given formulation and codeoptions and that you may want to record. However, the MATLAB and the Python client generate different tags for the same mathematical formulation.

16.2.2.4. Running a dumped problem

After you have dumped a problem formulation and at least one set of problem data, you can use either a matching pair of _F/_P files or any single _A file in JSON format to exactly reproduce your solver and problem instances. To do so, you need to perform the following two steps:

  1. Load problem formulation and data from JSON file or files calling:

[model, codeoptions, outputs, additionalData, problems] = ...
              ForcesLoadSymbolicDump( formulationFilename,problemFilenames );
model, options, outputs, additional, problems = \
              forcespro.dump.load(formulation_filename, problem_filename)

problemFilenames may either be a single file name or a cell array (list in Python) containing all the problem data set that you want to load. In case you have dumped both formulation and problem data set(s) at once within a single file, just pass that one as formulationFilename and do not specify problemFilenames. The returned problems variable is an array (list in Python) containing all problem sets found in the dumps.

  1. Re-generating and running the solver with dumped information by simply calling:

FORCES_NLP( model,codeoptions,outputs );
myFORCESsolver( problems(1) );
% and more problem instances if present
solver = model.generate_solver(codeoptions, outputs)
result, exitflag, info = my_forces_solver.solve(problems[1])
# and more problem instances if present

Tip

To get the filenames of dumped problems and formulations in a directory, simply use the function:

[formulationFilename, problemFilename] = ForcesFindDumpedProblems( tag,dumpDirectory );
formulation_filename, problem_filenames = forcespro.dump.find_problems(tag, dump_directory)

Both arguments are optional. If tag is not given, the function returns any dumped filenames regardless of their tag. The default dumpDirectory is the current working directory. problemFilenames is a cell array (list in Python) of problem filenames.

16.2.2.5. Limitations of symbolic dumps

Symbolic dumps only work with CasADi v3.5.1 for reasons beyond our control, which is why we currently do not plan to extend support to CasADi v2.4.2 or MathWorks’ Symbolic Math Toolbox.

16.2.3. Problem dumps from C

Problem dumps from C consider only the params struct containing the runtime parameters that are passed to the solver in each solver call. In order to save or load params structs, you can call the dump tool from any C script. This offers the opportunity to work directly on the embedded system. The data is stored in binary format. The problem dumps from C use the msgpack-c library.

Important

The problem dumps from C require dynamic memory allocation. This is because of the dependency on the msgpack-c library. Please check if your embedded platform supports dynamic memory allocation.

Before dumping problems from C for the first time, you need to install msgpack as described in Download and install msgpack for C. Then, the dumping procedure consists of the following three steps:

  1. Enabling the generation of the dump functions: When generating your solver, you need to set a codeoption in order to enable the generation of the dump functions.

  2. Dumping problem data: Call the generated serialize function in order to dump a params struct. Exactly one params struct can be stored at a time.

  3. Loading problem data: Call the generated deserialize function in order to load a params struct. Exactly one params struct can be stored at a time.

16.2.3.1. Download and install msgpack for C

Since the dump tool for problems from C requires msgpack, make sure you installed the library. You can either clone the github repository or you can simply download msgpack as a zip file. For the installation of the library, you need gcc >= 4.1.0 and cmake >= 2.8.1.

How to install:

  • Windows:

    1. Run cmake . in your terminal from the msgpack-c-c_master folder.

    2. Open the generated msgpack.sln file (located in the same folder) in Visual Studio and click Build. The generated library is located in msgpack-c-c_master/Debug.

  • Other platforms:

    1. Run cmake . in your terminal from the msgpack-c-c_master folder.

    2. Run make. The generated library is located in msgpack-c-c_master.

For more information about the installation of msgpack, see the README.md on github.

16.2.3.2. Enable the Generation of the Dump Functions

Before generating your solver you need to enable the generation of the dump functions. You can specify whether you want to dump from your host platform or/and from your target platform. For dumping from the host platform set:

codeoptions.serializeCParamsHost = 1;
codeoptions.serializeCParamsHost = 1

and for dumping from your specified target platform set:

codeoptions.serializeCParamsTarget = 1;
codeoptions.serializeCParamsTarget = 1

16.2.3.3. Dumping Problem Data

This section explains how to write and run a C script that dumps your problem data based on the high-level basic example (see High-level interface: Basic example). We assume your C script is in the same folder as your generated solver.

You can download the code of this example script to try it out for yourself by clicking here.

  1. Include your solver header called <solvername>.h. In the solver header, the <solvername>_params struct and the dumping routines <solvername>_serialize and <solvername>_deserialize are defined. Note that the params struct for binary problems is called <solvername>_binaryparams and for mixed integer problems it is <solvername>_integerparams.

#include "FORCESNLPsolver/include/FORCESNLPsolver.h"
  1. Create a params struct and fill it with your data:

/* create params struct */
FORCESNLPsolver_params params;

/* fill params struct with data */
params.xinit[0] = -4.;
params.xinit[1] = 2.;
for (int i = 0; i < 33; i++)
{
    params.x0[i] = 0.0;
}
  1. Choose a filename for the dump and call the serialization routine:

const char filename[] = "dump.msgpack";
int successSerialize = FORCESNLPsolver_serialize(&params, filename);
  1. Compile your script:

$ <Compiler_exec> my_C_dump_script.c <compiled_solver> -L<msgpack_lib_path> -l<msgpack_lib> <additional_libs>

Where:

  • <Compiler_exec> is your compiler (for example gcc)

  • my_C_dump_script.c is your script that calls the serialize function (for example serializationCParams_HighLevel_BasicExample.c)

  • <compiled_solver> is your compiled solver library (static or shared):

    • For Linux/MacOS/MinGW it is libFORCESNLPsolver.a or libFORCESNLPsolver.so in the lib or lib_target directory

    • For Windows it is FORCESNLPsolver_static.lib or FORCESNLPsolver.lib in the lib or lib_target directory

  • <msgpack_lib_path> specifies your path to the compiled msgpack library

    • For Linux/MacOS/MinGW it is the path to your msgpack-c-c_master folder

    • For Windows it is the path to your msgpack-c-c_master/Debug folder

  • <msgpack_lib> specifies the name of the compiled msgpack library

    • For the static library on Windows set it to msgpackc_import

    • Otherwise, it is msgpackc

  • <additional_libs> are possible libraries that need to be linked to resolve existing dependencies.

    • For Linux/MacOS it’s usually necessary to link the math library (-lm)

    • For Windows you usually need to link the iphlpapi.lib library (it’s distributed with the Intel Compiler, MinGW as well as Matlab) and sometimes some additional intel libraries (those are included in the FORCESPRO client under the folder libs_Intel – if missing they are downloaded after code generation)

The following shows how to compile the dump example script on Linux:

$ gcc serializationCParams_HighLevel_BasicExample.c FORCESNLPsolver/lib/libFORCESNLPsolver.so -L/path/to/msgpack-c -lmsgpackc -lm

16.2.3.4. Loading Problem Data

This section explains how to write and run a C script that loads a dumped C params struct based on the high-level basic example (see High-level interface: Basic example). We assume your C script is in the same folder as your generated solver.

You can download the code of this example script to try it out for yourself by clicking here.

  1. Include your solver header called <solvername>.h.

#include "FORCESNLPsolver/include/FORCESNLPsolver.h"
  1. Create an empty params struct:

FORCESNLPsolver_params dumped_params;
  1. Call the deserialization routine and pass the filename you chose when dumping the problem data:

int successDeserialize = FORCESNLPsolver_deserialize(&dumped_params, filename);
  1. Compile your script:

$ <Compiler_exec> my_C_dump_script.c <compiled_solver> -L<msgpack_lib_path> -l<msgpack_lib> <additional_libs>

Where:

  • <Compiler_exec> is your compiler (for example gcc)

  • my_C_dump_script.c is your script that calls the deserialize function (for example serializationCParams_HighLevel_BasicExample.c)

  • <compiled_solver> is your compiled solver library (static or shared):

    • For Linux/MacOS/MinGW it is libFORCESNLPsolver.a or libFORCESNLPsolver.so in the lib or lib_target directory

    • For Windows it is FORCESNLPsolver_static.lib or FORCESNLPsolver.lib in the lib or lib_target directory

  • <msgpack_lib_path> specifies your path to the compiled msgpack library

    • For Linux/MacOS/MinGW it is the path to your msgpack-c-c_master folder

    • For Windows it is the path to your msgpack-c-c_master/Debug folder

  • <msgpack_lib> specifies the name of the compiled msgpack library

    • For the static library on Windows set it to msgpackc_import

    • Otherwise, it is msgpackc

  • <additional_libs> are possible libraries that need to be linked to resolve existing dependencies.

    • For Linux/MacOS it’s usually necessary to link the math library (-lm)

    • For Windows you usually need to link the iphlpapi.lib library (it’s distributed with the Intel Compiler, MinGW as well as Matlab) and sometimes some additional intel libraries (those are included in the FORCESPRO client under the folder libs_Intel – if missing they are downloaded after code generation)

The following shows how to compile the dump example script on Linux:

$ gcc serializationCParams_HighLevel_BasicExample.c FORCESNLPsolver/lib/libFORCESNLPsolver.so -L/path/to/msgpack-c -lmsgpackc -lm