One of the key aspects in realizing large-scale fault-tolerant quantum computers is quantum error correction (QEC). The first essential step of QEC is to encode the logical state into physical qubits. However, there is no unique recipe for finding a compact quantum circuit that encodes or prepares the logical state in a fault-tolerant way, especially under hardware constraints such as qubit connectivity and gate set. In this work, we use reinforcement learning (RL) to automatically discover compact quantum circuits that prepare the logical state of a QEC code fault-tolerantly with the flag-based protocol for a given qubit connectivity and gate set. We first demonstrate that an RL agent can fault-tolerantly prepare logical states of a code with up to 15 physical qubits without any hardware constraints. We then show fault-tolerant logical state preparation on a 2D lattice connectivity with a smaller circuit size than using the standard transpilation technique to satisfy the hardware constraints.