Page 1 of 1

mwesd crashed with mwes.run invoked oom-killer

Posted: Fri Jul 21, 2017 4:13 am
by cwh168998
the mwesd run for couple of months with no this issue. But since Monday, It begins to crash with error:
2017-07-20T09:46:25.212960-04:00 mwes kernel: [158858.125625] mwes.run invoked oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0
2017-07-20T09:56:41.588929-04:00 mwes kernel: [159474.505747] mwes.run invoked oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0
2017-07-20T10:54:34.589010-04:00 mwes kernel: [ 2652.791936] mwes.run invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
2017-07-20T11:49:55.189778-04:00 mwes kernel: [ 5973.383037] mwes.run invoked oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0

It looks like it run out of memory.2017-07-20T12:20:47.535178-04:00 mwes kernel: [ 1227.282383] dmeventd invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=-1000
2017-07-20T12:20:47.574128-04:00 mwes kernel: [ 1227.282387] dmeventd cpuset=/ mems_allowed=0
2017-07-20T12:20:47.574135-04:00 mwes kernel: [ 1227.282390] CPU: 1 PID: 384 Comm: dmeventd Tainted: G X 3.12.28-4-default #1
2017-07-20T12:20:47.574138-04:00 mwes kernel: [ 1227.282391] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS Hyper-V UEFI Release v1.0 11/26/2012
2017-07-20T12:20:47.574140-04:00 mwes kernel: [ 1227.282393] 0000000000000000 ffffffff8150b1db ffff880016cca280 ffffffff815069de
2017-07-20T12:20:47.574141-04:00 mwes kernel: [ 1227.282396] 0000000000000000 0000000000000000 0000000000000000 ffff88001f10ad80
2017-07-20T12:20:47.574143-04:00 mwes kernel: [ 1227.282398] 0000000000000000 0000000000000000 0000000000000000 0000000000000000
2017-07-20T12:20:47.574145-04:00 mwes kernel: [ 1227.282400] Call Trace:
2017-07-20T12:20:47.574147-04:00 mwes kernel: [ 1227.282412] [<ffffffff8100467d>] dump_trace+0x7d/0x2d0
2017-07-20T12:20:47.574149-04:00 mwes kernel: [ 1227.282416] [<ffffffff81004964>] show_stack_log_lvl+0x94/0x170
2017-07-20T12:20:47.574150-04:00 mwes kernel: [ 1227.282420] [<ffffffff81005d91>] show_stack+0x21/0x50
2017-07-20T12:20:47.574467-04:00 mwes kernel: [ 1227.282424] [<ffffffff8150b1db>] dump_stack+0x41/0x51
2017-07-20T12:20:47.574473-04:00 mwes kernel: [ 1227.282428] [<ffffffff815069de>] dump_header+0x76/0x1e8
2017-07-20T12:20:47.574475-04:00 mwes kernel: [ 1227.282434] [<ffffffff81138eea>] oom_kill_process+0x21a/0x370
2017-07-20T12:20:47.574476-04:00 mwes kernel: [ 1227.282441] [<ffffffff81139688>] out_of_memory+0x4a8/0x4e0
2017-07-20T12:20:47.574478-04:00 mwes kernel: [ 1227.282445] [<ffffffff8150753d>] __alloc_pages_slowpath+0x632/0x796
2017-07-20T12:20:47.574479-04:00 mwes kernel: [ 1227.282448] [<ffffffff8113ee7a>] __alloc_pages_nodemask+0x1ea/0x210
2017-07-20T12:20:47.574481-04:00 mwes kernel: [ 1227.282454] [<ffffffff8117bc83>] alloc_pages_current+0xa3/0x170
2017-07-20T12:20:47.574651-04:00 mwes kernel: [ 1227.282458] [<ffffffff81137c6d>] filemap_fault+0x17d/0x3f0
2017-07-20T12:20:47.574655-04:00 mwes kernel: [ 1227.282463] [<ffffffff8115b005>] __do_fault+0x75/0x500
2017-07-20T12:20:47.574657-04:00 mwes kernel: [ 1227.282466] [<ffffffff8115e937>] handle_mm_fault+0x447/0xe10
2017-07-20T12:20:47.574658-04:00 mwes kernel: [ 1227.282470] [<ffffffff81514b1a>] __do_page_fault+0x14a/0x4f0
2017-07-20T12:20:47.574660-04:00 mwes kernel: [ 1227.282475] [<ffffffff81511888>] page_fault+0x28/0x30
2017-07-20T12:20:47.574661-04:00 mwes kernel: [ 1227.282491] [<00007f55a1f71a43>] 0x7f55a1f71a42
2017-07-20T12:20:47.574663-04:00 mwes kernel: [ 1227.282493] Mem-Info:
2017-07-20T12:20:47.574827-04:00 mwes kernel: [ 1227.282494] Node 0 DMA per-cpu:
2017-07-20T12:20:47.576556-04:00 mwes kernel: [ 1227.282496] CPU 0: hi: 0, btch: 1 usd: 0
2017-07-20T12:20:47.576559-04:00 mwes kernel: [ 1227.282497] CPU 1: hi: 0, btch: 1 usd: 0
2017-07-20T12:20:47.576561-04:00 mwes kernel: [ 1227.282498] CPU 2: hi: 0, btch: 1 usd: 0
2017-07-20T12:20:47.576562-04:00 mwes kernel: [ 1227.282499] CPU 3: hi: 0, btch: 1 usd: 0
2017-07-20T12:20:47.576564-04:00 mwes kernel: [ 1227.282500] Node 0 DMA32 per-cpu:
2017-07-20T12:20:47.576565-04:00 mwes kernel: [ 1227.282502] CPU 0: hi: 186, btch: 31 usd: 1
2017-07-20T12:20:47.576805-04:00 mwes kernel: [ 1227.282503] CPU 1: hi: 186, btch: 31 usd: 1
2017-07-20T12:20:47.576809-04:00 mwes kernel: [ 1227.282504] CPU 2: hi: 186, btch: 31 usd: 0
2017-07-20T12:20:47.576811-04:00 mwes kernel: [ 1227.282505] CPU 3: hi: 186, btch: 31 usd: 30
2017-07-20T12:20:47.576812-04:00 mwes kernel: [ 1227.282506] Node 0 Normal per-cpu:
2017-07-20T12:20:47.576814-04:00 mwes kernel: [ 1227.282507] CPU 0: hi: 186, btch: 31 usd: 4
2017-07-20T12:20:47.576815-04:00 mwes kernel: [ 1227.282508] CPU 1: hi: 186, btch: 31 usd: 169
2017-07-20T12:20:47.578209-04:00 mwes kernel: [ 1227.282509] CPU 2: hi: 186, btch: 31 usd: 0
2017-07-20T12:20:47.578214-04:00 mwes kernel: [ 1227.282510] CPU 3: hi: 186, btch: 31 usd: 55
2017-07-20T12:20:47.578216-04:00 mwes kernel: [ 1227.282514] active_anon:3833 inactive_anon:3815 isolated_anon:0
2017-07-20T12:20:47.578217-04:00 mwes kernel: [ 1227.282514] active_file:5913 inactive_file:5761 isolated_file:32
2017-07-20T12:20:47.578219-04:00 mwes kernel: [ 1227.282514] unevictable:20 dirty:7940 writeback:156 unstable:0
2017-07-20T12:20:47.578221-04:00 mwes kernel: [ 1227.282514] free:10609 slab_reclaimable:7519 slab_unreclaimable:5275
2017-07-20T12:20:47.578222-04:00 mwes kernel: [ 1227.282514] mapped:322 shmem:221 pagetables:903 bounce:0
2017-07-20T12:20:47.578226-04:00 mwes kernel: [ 1227.282514] free_cma:0
2017-07-20T12:20:47.578228-04:00 mwes kernel: [ 1227.282517] Node 0 DMA free:12292kB min:24kB low:28kB high:36kB active_anon:0kB inactive_anon:0kB active_file:12kB inactive_file:16kB unevictable:4kB isolated(anon):0kB isolated(file):0kB present:15996kB managed:15908kB mlocked:4kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:276kB slab_unreclaimable:508kB kernel_stack:24kB pagetables:68kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:5792 all_unreclaimable? yes
2017-07-20T12:20:47.578230-04:00 mwes kernel: [ 1227.282521] lowmem_reserve[]: 0 464 5638 5638
2017-07-20T12:20:47.578232-04:00 mwes kernel: [ 1227.282523] Node 0 DMA32 free:21456kB min:788kB low:984kB high:1180kB active_anon:4kB inactive_anon:0kB active_file:1992kB inactive_file:2244kB unevictable:76kB isolated(anon):0kB isolated(file):0kB present:505256kB managed:475884kB mlocked:76kB dirty:2416kB writeback:0kB mapped:276kB shmem:28kB slab_reclaimable:16712kB slab_unreclaimable:18364kB kernel_stack:1760kB pagetables:2344kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:37392 all_unreclaimable? yes
2017-07-20T12:20:47.578234-04:00 mwes kernel: [ 1227.282527] lowmem_reserve[]: 0 0 5174 5174
2017-07-20T12:20:47.578239-04:00 mwes kernel: [ 1227.282529] Node 0 Normal free:8688kB min:8792kB low:10988kB high:13188kB active_anon:15328kB inactive_anon:15284kB active_file:21648kB inactive_file:20784kB unevictable:0kB isolated(anon):0kB isolated(file):128kB present:6815744kB managed:5298176kB mlocked:0kB dirty:29344kB writeback:644kB mapped:1012kB shmem:856kB slab_reclaimable:13088kB slab_unreclaimable:2228kB kernel_stack:112kB pagetables:1200kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:441636 all_unreclaimable? yes
2017-07-20T12:20:47.578241-04:00 mwes kernel: [ 1227.282533] lowmem_reserve[]: 0 0 0 0
2017-07-20T12:20:47.578242-04:00 mwes kernel: [ 1227.282535] Node 0 DMA: 80*4kB (UM) 49*8kB (UM) 28*16kB (UM) 20*32kB (UM) 14*64kB (UM) 11*128kB (UM) 6*256kB (UM) 3*512kB (M) 1*1024kB (M) 2*2048kB (ER) 0*4096kB = 12296kB
2017-07-20T12:20:47.578245-04:00 mwes kernel: [ 1227.282545] Node 0 DMA32: 4991*4kB (M) 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 1*512kB (R) 1*1024kB (R) 0*2048kB 0*4096kB = 21500kB
2017-07-20T12:20:47.578246-04:00 mwes kernel: [ 1227.282551] Node 0 Normal: 2282*4kB (M) 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 9128kB
2017-07-20T12:20:47.578248-04:00 mwes kernel: [ 1227.282558] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
2017-07-20T12:20:47.578712-04:00 mwes kernel: [ 1227.282559] 12898 total pagecache pages
2017-07-20T12:20:47.578717-04:00 mwes kernel: [ 1227.282560] 1110 pages in swap cache
2017-07-20T12:20:47.578718-04:00 mwes kernel: [ 1227.282562] Swap cache stats: add 906442, delete 905332, find 100163/140471
2017-07-20T12:20:47.578720-04:00 mwes kernel: [ 1227.282563] Free swap = 2033268kB
2017-07-20T12:20:47.578722-04:00 mwes kernel: [ 1227.282564] Total swap = 2104316kB
2017-07-20T12:20:47.578723-04:00 mwes kernel: [ 1227.282565] 1834249 pages RAM
2017-07-20T12:20:47.578725-04:00 mwes kernel: [ 1227.282565] 0 pages HighMem/MovableOnly
2017-07-20T12:20:47.579008-04:00 mwes kernel: [ 1227.282566] 379392 pages reserved
2017-07-20T12:20:47.579012-04:00 mwes kernel: [ 1227.282567] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
2017-07-20T12:20:47.579014-04:00 mwes kernel: [ 1227.282573] [ 373] 0 373 7764 55 20 82 0 systemd-journal
2017-07-20T12:20:47.579016-04:00 mwes kernel: [ 1227.282575] [ 384] 0 384 5997 0 15 54 -1000 dmeventd
2017-07-20T12:20:47.579017-04:00 mwes kernel: [ 1227.282577] [ 398] 0 398 10670 19 24 239 -1000 systemd-udevd
2017-07-20T12:20:47.579019-04:00 mwes kernel: [ 1227.282579] [ 401] 0 401 3008 0 10 800 0 haveged
2017-07-20T12:20:47.579023-04:00 mwes kernel: [ 1227.282582] [ 695] 499 695 10637 116 25 116 -900 dbus-daemon
2017-07-20T12:20:47.579025-04:00 mwes kernel: [ 1227.282583] [ 701] 0 701 4794 26 14 36 0 irqbalance
2017-07-20T12:20:47.579026-04:00 mwes kernel: [ 1227.282585] [ 706] 496 706 183690 37 39 88 0 nscd
2017-07-20T12:20:47.579028-04:00 mwes kernel: [ 1227.282587] [ 735] 0 735 84451 0 33 245 0 rsyslogd
2017-07-20T12:20:47.579029-04:00 mwes kernel: [ 1227.282589] [ 743] 0 743 5023 48 15 47 0 systemd-logind
2017-07-20T12:20:47.579030-04:00 mwes kernel: [ 1227.282591] [ 748] 1001 748 284826 319 120 4328 0 mwes.run
2017-07-20T12:20:47.579032-04:00 mwes kernel: [ 1227.282593] [ 809] 0 809 7290 9 19 214 0 wickedd-auto4
2017-07-20T12:20:47.580106-04:00 mwes kernel: [ 1227.282594] [ 810] 0 810 7307 9 21 217 0 wickedd-dhcp4
2017-07-20T12:20:47.580111-04:00 mwes kernel: [ 1227.282596] [ 811] 0 811 7313 9 19 216 0 wickedd-dhcp6
2017-07-20T12:20:47.580113-04:00 mwes kernel: [ 1227.282598] [ 813] 0 813 7292 5 21 240 0 wickedd
2017-07-20T12:20:47.580115-04:00 mwes kernel: [ 1227.282599] [ 814] 0 814 7172 0 19 91 0 wickedd-nanny
2017-07-20T12:20:47.580116-04:00 mwes kernel: [ 1227.282601] [ 1235] 0 1235 4704 15 13 133 0 cron
2017-07-20T12:20:47.580118-04:00 mwes kernel: [ 1227.282603] [ 1237] 0 1237 2188 0 9 42 0 xinetd
2017-07-20T12:20:47.580119-04:00 mwes kernel: [ 1227.282605] [ 1259] 0 1259 11773 0 26 156 -1000 sshd
2017-07-20T12:20:47.580123-04:00 mwes kernel: [ 1227.282607] [ 1546] 0 1546 1110 1 8 25 0 agetty
2017-07-20T12:20:47.580125-04:00 mwes kernel: [ 1227.282608] [ 1552] 0 1552 21913 17 43 219 0 sshd
2017-07-20T12:20:47.580127-04:00 mwes kernel: [ 1227.282610] [ 1557] 0 1557 21860 0 43 225 0 sshd
2017-07-20T12:20:47.580128-04:00 mwes kernel: [ 1227.282612] [ 1560] 0 1560 3182 0 10 75 0 sftp-server
2017-07-20T12:20:47.580130-04:00 mwes kernel: [ 1227.282614] [ 1588] 0 1588 3742 117 12 398 0 bash
2017-07-20T12:20:47.580131-04:00 mwes kernel: [ 1227.282615] [ 2067] 0 2067 289966 4960 174 6310 0 mwes.run
2017-07-20T12:20:47.580702-04:00 mwes kernel: [ 1227.282618] [ 2931] 0 2931 21913 46 46 221 0 sshd
2017-07-20T12:20:47.580706-04:00 mwes kernel: [ 1227.282619] [ 2934] 0 2934 21860 7 45 219 0 sshd
2017-07-20T12:20:47.580720-04:00 mwes kernel: [ 1227.282621] [ 2938] 0 2938 3235 22 11 61 0 sftp-server
2017-07-20T12:20:47.580722-04:00 mwes kernel: [ 1227.282623] [ 2950] 0 2950 3709 30 12 457 0 bash
2017-07-20T12:20:47.580723-04:00 mwes kernel: [ 1227.282625] [ 3016] 0 3016 3519 51 13 119 0 top
2017-07-20T12:20:47.580725-04:00 mwes kernel: [ 1227.282626] Out of memory: Kill process 2067 (mwes.run) score 5 or sacrifice child
2017-07-20T12:20:47.580727-04:00 mwes kernel: [ 1227.282629] Killed process 2067 (mwes.run) total-vm:1159864kB, anon-rss:19816kB, file-rss:24kB


Please help

Re: mwesd crashed with mwes.run invoked oom-killer

Posted: Fri Jul 21, 2017 10:20 am
by nick.bolton
Hi

We'd need to see the debug logs from var\log\mwes

Re: mwesd crashed with mwes.run invoked oom-killer

Posted: Thu Jul 27, 2017 2:14 am
by cwh168998
MWESD is installed in a virtual Linux machine. I believe that Hyper-V 2012 R2 updated silently. After updated, Hyper-V 2012 memory management does not work well with Linux VM. I increased minimum memory to maximum memory. MWESD works without crash since then.

Thank you for the help.

Re: mwesd crashed with mwes.run invoked oom-killer

Posted: Thu Jul 27, 2017 7:09 am
by nick.bolton
Thanks for letting us know!