mongodb - ERROR: child process failed, exited with error number 100

Question

1. MongoDB is configured in replica set mode and started successfully
2. Insert data into it. Both master and slave2 are normal, but slave1 crashes abnormally
3. Restart slave1 and the error is as follows

about to fork child process, waiting until server is ready for connections.
forked process: 756411
ERROR: child process failed, exited with error number 1

4. After deleting mongo.lock and mongodb.log and starting in repair mode, the error is as follows

about to fork child process, waiting until server is ready for connections.
forked process: 756411
ERROR: child process failed, exited with error number 100

5. Check the content of mongodb.log as follows, and a bunch of No space left on device was reported

2017-02-28T15:22:10.933+0800 E STORAGE  [repl writer worker 3] WiredTiger (28) [1488266530:932986][21657:0x7f903153f700], WT_SESSION.commit_transaction: journal/WiredTigerLog.0000000003: ha
ndle-write: pwrite: failed to write 45952 bytes at offset 8572928: No space left on device
2017-02-28T15:22:10.933+0800 I -        [repl writer worker 3] Invariant failure: s->commit_transaction(s, NULL) resulted in status UnknownError: 28: No space left on device at src/mongo/db
/storage/wiredtiger/wiredtiger_recovery_unit.cpp 213
2017-02-28T15:22:10.933+0800 E STORAGE  [repl writer worker 0] WiredTiger (28) [1488266530:933845][21657:0x7f9032d42700], WT_SESSION.commit_transaction: journal/WiredTigerLog.0000000003: ha
ndle-write: pwrite: failed to write 129408 bytes at offset 8753152: No space left on device
2017-02-28T15:22:10.933+0800 I -        [repl writer worker 0] Invariant failure: s->commit_transaction(s, NULL) resulted in status UnknownError: 28: No space left on device at src/mongo/db
/storage/wiredtiger/wiredtiger_recovery_unit.cpp 213
2017-02-28T15:22:10.934+0800 E STORAGE  [repl writer worker 7] WiredTiger (28) [1488266530:934510][21657:0x7f902f53b700], WT_SESSION.commit_transaction: journal/WiredTigerLog.0000000003: ha
ndle-write: pwrite: failed to write 131840 bytes at offset 8882560: No space left on device
2017-02-28T15:22:10.934+0800 I -        [repl writer worker 7] Invariant failure: s->commit_transaction(s, NULL) resulted in status UnknownError: 28: No space left on device at src/mongo/db
/storage/wiredtiger/wiredtiger_recovery_unit.cpp 213
2017-02-28T15:22:10.935+0800 E STORAGE  [repl writer worker 1] WiredTiger (28) [1488266530:935197][21657:0x7f9032541700], WT_SESSION.commit_transaction: journal/WiredTigerLog.0000000003: ha
ndle-write: pwrite: failed to write 131840 bytes at offset 9014400: No space left on device
2017-02-28T15:22:10.935+0800 I -        [repl writer worker 1] Invariant failure: s->commit_transaction(s, NULL) resulted in status UnknownError: 28: No space left on device at src/mongo/db
/storage/wiredtiger/wiredtiger_recovery_unit.cpp 213
2017-02-28T15:22:10.936+0800 E STORAGE  [repl writer worker 9] WiredTiger (28) [1488266530:936508][21657:0x7f902e539700], WT_SESSION.commit_transaction: journal/

6. Has anyone encountered similar problems? Still looking forward to the answer

Answer 1

Your disk space is insufficient, causing log submission to fail. df -hCheck the disk space to see if the disk corresponding to /mongo/db
/storage/wiredtiger/ is full. You can usually delete the mongo log file to free up some disk space.