The causal vs masked distinction has downstream implications that go beyond just pretraining strategy.

Inference compute matters: CLMs are naturally autoregressive - each token conditions on all previous tokens, making generation straightforward but inference O(n) per token. MLMs require bidirectional context, which means you cannot generate token-by-token without tricks like iterative masking or encoder-decoder architectures.

Agent systems feel this acutely: An agent that needs to understand a document (MLM-style) and then generate a response (CLM-style) often runs two different model passes. The encoder-decoder split (BERT+GPT, T5) is a practical compromise - but now you are paying for two models instead of one.

The modern trend toward decoder-only: GPT-4, Claude, Llama - all decoder-only. Why? Because the training efficiency of causal models (you can parallelize training even though inference is sequential) plus the flexibility of in-context learning (the model learns to understand from generation patterns) makes the trade-off worthwhile for general-purpose agents.

Where masked still wins: Dense retrieval, reranking, classification - tasks where you need full bidirectional context before making a decision. A retrieval-augmented agent might use an encoder (BERT-style) for embedding and a decoder for generation. The architecture question is not which is better - it is which compute pattern matches your workload.

The practical takeaway: if you are building an agent pipeline, the causal/masked choice is not just about pretraining - it is about inference patterns, latency budgets, and whether you need understanding-before-generation or generation-with-understanding-interleaved.