[Cbe-oss-dev] Hard crash on fb write past 17MB

[Geert Uytterhoeven]

	Hi Mike,

On Tue, 19 Dec 2006, Mike Acton wrote:
> Having a problem which causes a hard crash of the OS:
> 
> I can mmap 18MB from /dev/fb0 (whatever configuration: 1MB x 18, 18MB
> x 1, etc.) without errors. (Trying to map more than 18MB is rejected
> by mmap)
> 
> Writing in the range 0-17MB causes no problems.
> Writing to the range 17-18MB causes the system to stop without any
> apparent feedback.
> Reading the range 0-18MB cases no problems.
> 
> I have a small test program which demonstrates the problem:
> http://www.cellperformance.com/public/attachments/test_fb_alloc.c
> 
> To compile: gcc -std=c99 test_fb_alloc.c -o test_fb_alloc
> 
> [It will prompt before the offending bit to give you a chance to skip it.]
> 
> This is on YDL5 for PS3 with kernel 2.6.16
> 
> Anyone have an idea?

I think I found out what's happening...

The PS3 frame buffer device uses a virtual frame buffer in main memory. 
It has a block of 19 MiB allocated in bss (2.6.16) or bootmem (2.6.20-rc*),
which is then aligned to the next MiB boundary to yield an aligned buffer of 18
MiB.

However, the actual virtual frame buffer is not the beginning of this buffer of
18 MiB! For both fb_info.screen_base and ps3fb_mmap(), an offset (FB_OFF(i) +
VP_OFF(i)) is added. This offset is always a non-zero multiple of 64 kiB.

As both fb_write() and ps3fb_mmap() assume a buffer limit based on the actual
frame buffer size (18 MiB) and don't take the additional offset into account,
they allow to write a few multiples of 64 kiB beyond the end of the 18 MiB
buffer. Depending on the actual position of the aligned 18 MiB buffer inside
the initial 19 MiB buffer, you may be able to write beyond the end of the 19
MiB buffer.  This will overwrite innocent kernel memory, and may cause a crash.

Gr{oetje,eeting}s,

						Geert

--
Geert Uytterhoeven -- Sony Network and Software Technology Center Europe (NSCE)
Geert.Uytterhoeven at sonycom.com ------- The Corporate Village, Da Vincilaan 7-D1
Voice +32-2-7008453 Fax +32-2-7008622 ---------------- B-1935 Zaventem, Belgium