Re: nginx застрял в sched_yield()

[Denis F. Latypoff <latypoff@xxxxxxxxx>]

31.03.2011, 16:30, "Maxim Dounin" <mdounin@xxxxxxxxxx>:
> Hello!
>
> On Thu, Mar 31, 2011 at 04:44:41AM +0400, Denis F. Latypoff wrote:
>
>>  Здрасьте, наконец-то отловил его, уже такое бывало, но
>>  на продакшене, по этому не получалось бэктрейс глянуть, надо было
>>  быстро все починить. Проявляется при HUP'е:
>>
>>  (gdb) bt
>>  #0  0x0000003744abac77 in sched_yield () from /lib64/libc.so.6
>>  #1  0x0000000000413667 in ngx_spinlock (lock=0x2b1216ba3000, value=14637,
>>      spin=1024) at src/core/ngx_spinlock.c:39
>
> fr 1
> p ngx_ncpu
> p *lock
>
> Последнее напечатает pid процесса, про него тоже желательно узнать
> подробности.

(gdb) fr 1
#1  0x0000000000413667 in ngx_spinlock (lock=0x2b1216ba3000, value=14637,
    spin=1024) at src/core/ngx_spinlock.c:39
39              ngx_sched_yield();
(gdb) p ngx_ncpu
$3 = 1
(gdb) fr 1
#1  0x0000000000413667 in ngx_spinlock (lock=0x2b1216ba3000, value=14637,
    spin=1024) at src/core/ngx_spinlock.c:39
39              ngx_sched_yield();
(gdb) p ngx_ncpu
$4 = 1
(gdb) p *lock
$5 = 15792

>
> Можно ещё там же походить и убедиться, что оно на самом деле
> крутится в ngx_spinlock(), а не висит в sched_yield().

Да чо ходить, вон оно http://gostats.org/localhost-cpu-day.png

>
>>  #2  0x00000000004452ad in ngx_http_file_cache_expire (
>>      data=<value optimized out>) at src/http/ngx_http_file_cache.c:1076
>>  #3  ngx_http_file_cache_manager (data=<value optimized out>)
>>      at src/http/ngx_http_file_cache.c:1196
>>  #4  0x000000000041fdba in ngx_cache_manager_process_handler 
>> (ev=0x7ffff2bee870)
>>      at src/os/unix/ngx_process_cycle.c:1346
>>  #5  0x000000000041a299 in ngx_event_expire_timers ()
>>      at src/event/ngx_event_timer.c:149
>>  #6  0x000000000041a1c5 in ngx_process_events_and_timers (cycle=0x75f4fb0)
>>      at src/event/ngx_event.c:261
>>  #7  0x0000000000420a88 in ngx_cache_manager_process_cycle (cycle=0x75f4fb0,
>>      data=<value optimized out>) at src/os/unix/ngx_process_cycle.c:1328
>>  #8  0x000000000041e5c7 in ngx_spawn_process (cycle=0x75f4fb0,
>>      proc=0x420919 <ngx_cache_manager_process_cycle>, data=0x67d400,
>>      name=0x46803f "cache manager process", respawn=-4)
>>      at src/os/unix/ngx_process.c:196
>>  #9  0x000000000041f21a in ngx_start_cache_manager_processes 
>> (cycle=0x75f4fb0,
>>      respawn=1) at src/os/unix/ngx_process_cycle.c:398
>>  #10 0x00000000004207ee in ngx_master_process_cycle (cycle=0x75f4fb0)
>>      at src/os/unix/ngx_process_cycle.c:251
>>  #11 0x0000000000406b8a in main (argc=12, argv=0x75c9ae9)
>>      at src/core/nginx.c:405
>>
>>  Процесс пока не убиваю, сервер тестовый,
>>  если надо структурки посмотреть, то пожалуйста.
>>
>>  [root@gostats ~]# nginx -V
>>  nginx: nginx version: nginx/0.9.6
>>  nginx: built by gcc 4.1.2 20080704 (Red Hat 4.1.2-48)
>
> Подозрительно знакомая версия gcc...
>
> Ещё пожалуйста
>
> disass ngx_spinlock

(gdb) disass ngx_spinlock
Dump of assembler code for function ngx_spinlock:
0x0000000000413600 <ngx_spinlock+0>:    push   %r12
0x0000000000413602 <ngx_spinlock+2>:    push   %rbp
0x0000000000413603 <ngx_spinlock+3>:    push   %rbx
0x0000000000413604 <ngx_spinlock+4>:    mov    %rdi,%rbx
0x0000000000413607 <ngx_spinlock+7>:    mov    %rsi,%rbp
0x000000000041360a <ngx_spinlock+10>:   mov    %rdx,%r12
0x000000000041360d <ngx_spinlock+13>:   mov    (%rbx),%rax
0x0000000000413610 <ngx_spinlock+16>:   test   %rax,%rax
0x0000000000413613 <ngx_spinlock+19>:   jne    0x41361c <ngx_spinlock+28>
0x0000000000413615 <ngx_spinlock+21>:   lock cmpxchg %rbp,(%rbx)
0x000000000041361a <ngx_spinlock+26>:   je     0x413669 <ngx_spinlock+105>
0x000000000041361c <ngx_spinlock+28>:   cmpq   $0x1,0x285e14(%rip)        # 
0x699438 <ngx_ncpu>
0x0000000000413624 <ngx_spinlock+36>:   jle    0x413662 <ngx_spinlock+98>
0x0000000000413626 <ngx_spinlock+38>:   mov    $0x1,%edx
0x000000000041362b <ngx_spinlock+43>:   cmp    $0x1,%r12
0x000000000041362f <ngx_spinlock+47>:   ja     0x413655 <ngx_spinlock+85>
0x0000000000413631 <ngx_spinlock+49>:   jmp    0x413662 <ngx_spinlock+98>
0x0000000000413633 <ngx_spinlock+51>:   pause
0x0000000000413635 <ngx_spinlock+53>:   add    $0x1,%rax
0x0000000000413639 <ngx_spinlock+57>:   cmp    %rdx,%rax
0x000000000041363c <ngx_spinlock+60>:   jne    0x413633 <ngx_spinlock+51>
0x000000000041363e <ngx_spinlock+62>:   mov    (%rbx),%rax
0x0000000000413641 <ngx_spinlock+65>:   test   %rax,%rax
0x0000000000413644 <ngx_spinlock+68>:   jne    0x41364d <ngx_spinlock+77>
0x0000000000413646 <ngx_spinlock+70>:   lock cmpxchg %rbp,(%rbx)
0x000000000041364b <ngx_spinlock+75>:   je     0x413669 <ngx_spinlock+105>
0x000000000041364d <ngx_spinlock+77>:   add    %rdx,%rdx
0x0000000000413650 <ngx_spinlock+80>:   cmp    %rdx,%r12
0x0000000000413653 <ngx_spinlock+83>:   jbe    0x413662 <ngx_spinlock+98>
0x0000000000413655 <ngx_spinlock+85>:   mov    $0x0,%eax
0x000000000041365a <ngx_spinlock+90>:   test   %rdx,%rdx
0x000000000041365d <ngx_spinlock+93>:   jne    0x413633 <ngx_spinlock+51>
0x000000000041365f <ngx_spinlock+95>:   nop
0x0000000000413660 <ngx_spinlock+96>:   jmp    0x41363e <ngx_spinlock+62>
0x0000000000413662 <ngx_spinlock+98>:   callq  0x404750 <sched_yield@plt>
0x0000000000413667 <ngx_spinlock+103>:  jmp    0x41360d <ngx_spinlock+13>
0x0000000000413669 <ngx_spinlock+105>:  pop    %rbx
0x000000000041366a <ngx_spinlock+106>:  pop    %rbp
0x000000000041366b <ngx_spinlock+107>:  pop    %r12
0x000000000041366d <ngx_spinlock+109>:  nopl   (%rax)
0x0000000000413670 <ngx_spinlock+112>:  retq
End of assembler dump.

> disass ngx_atomic_cmp_set

gdb сказало, что нету такого.

>
>>  nginx: TLS SNI support disabled
>>  nginx: configure arguments: --prefix=/opt/nginx --user=nobody 
>> --group=nobody --with-http_image_filter_module 
>> --with-http_stub_status_module --with-http_ssl_module 
>> --with-http_xslt_module --with-http_geoip_module
>>
>>  [root@gostats ~]# uname -a
>>  Linux gostats.org 2.6.18-194.26.1.el5 #1 SMP Tue Nov 9 12:54:20 EST 2010 
>> x86_64 x86_64 x86_64 GNU/Linux
>
> Maxim Dounin
>

-- 
br, Denis F. Latypoff.

_______________________________________________
nginx-ru mailing list
nginx-ru@xxxxxxxxx
http://nginx.org/mailman/listinfo/nginx-ru