Formulating an optimization problem the right way often matters more than the algorithm that solves it, but it remains an art that demands expertise. When we move into black‑box optimization territory — where the objective function cannot be expressed in closed form and only pointwise values are observed — search space design becomes the real bottleneck. That is where the new BBOWP‑Bench benchmark tries to understand whether Large Language Models can help, or at least learn to do so.

The paper introduces Black‑Box Optimization Word Problems, a setting in which a system — typically an LLM — must infer both the search space and the optimization algorithm from a textual description of the task. This is not an abstract exercise: in industrial practice, an engineer tuning the parameters of a chemical process or optimizing a materials recipe faces exactly this situation, with the added burden that trials cost time and money and the data are often sensitive. Knowing whether an LLM can translate a written brief into an operational optimization strategy means assessing whether we can keep certain activities in‑house, without exposing descriptions that contain proprietary know‑how.

The benchmark pairs each instance with a natural‑language description, an executable environment, and a human‑designed baseline formulation. Initial experiments show that LLMs handle algorithm selection surprisingly well, adapting to the available evaluation budget: when the budget is low they prefer greedier methods, while with a larger budget they propose evolutionary or Bayesian algorithms. Here the models demonstrate that they have internalized a fair amount of engineering intuition about the cost‑quality trade‑off.

The weak spot is search space design. When the problem description is generic or deliberately sparse, LLMs struggle to identify which variables are truly relevant and to set plausible bounds for them. This limitation is not unexpected: the domain knowledge needed to define a sensible search neighborhood is often tacit, rooted in an individual expert’s experience, and current models cannot yet fill that gap from text alone.

For those building on‑premise optimization pipelines the message is twofold. On one hand, the ability to select the algorithm automatically can reduce reliance on specialists, especially when companies want to standardize procedures without resorting to external consultants. On the other, the fragility in search space design calls for caution: an LLM suggesting inadequate ranges can waste compute resources or, worse, make the process converge to unacceptable local optima. In environments where data must remain physically on corporate servers for sovereignty reasons, every decision delegated to the model has to be checked against safety and reproducibility metrics.

BBOWP‑Bench does not — and could not — offer a turnkey solution. It works instead as an evaluation framework that, by measuring performance across a range of different tasks, lets us gauge how mature the automation of black‑box problem formulation really is. The fact that code and dataset are public helps anyone wanting to test self‑hosted models or models fine‑tuned on specific domains. Looking ahead, as LLMs improve their handling of contextual knowledge, we will see these tools progressively integrated into local R&D workflows where black‑box optimization is everyday bread and butter but where confidentiality remains a non‑negotiable architectural constraint.