C problem...se megethos domhs

[Aggelos Economopoulos]

On Sun, 25 Jan 2004 20:08:55 +0200
"John Tsiombikas (Nuclear / the Lab)" <nuclear at siggraph.org> wrote:

> Aggelos Economopoulos wrote:
> >>to file i/o einai buffered (an xrisimopoieis fwrite/fread klp) eite
> >>kaneis ena fwrite() olo to struct mazi, eite kaneis 10 mikra
> >fwrite()>gia kathe member ksexorista, to performance einai idio.
> > 
> > Elpizw na ennoeis /peripou/ idio, giati to performance katafanws
> > *den* einai idio. An mia struct apoteleitai apo (p.x.) 10 32-bit
> > integers, katalhgeis na kaneis 10 fwrite() (dhladh ta dekaplasia
> > function calls) apla gia na doulepsei h stdio wste na ta balei pali
> > se enan buffer kai na kalesei thn read(). Profanws se ena
> > sunh8hsmeno programma, opou to i/o den einai to bottleneck, auto den
> > prokeitai na epiferei metrhsimo slowdown, an omws h berkleydb kai h
> > mysql xrhsimopoiousan apokleistika thn stdio 8a to eixan kleisei to
> > magazi.
> 
> Koita na deis otan milame gia read/write se disko pou einai inherently
> argi diadikasia to function call overhead einai pragmatika ameliteo,
> eite ginoun 10 function calls eite 100 den yparxei ousiastiki diafora.
> to bottleneck einai stin diadikasia tou I/O.

Ektos tou oti to na spatalas cpu einai impolite otan mporei na trexoun
alla threads (akoma kai *dika* sou threads, sto idio process), to
write() ligo exei na kanei me to I/O - epistrefei otan antigrapsei ta
periexomena tou buffer sthn in-memory cache tou leitourgikou. To pote
8a graftei to block sto disko e3artatai apo to th vm pressure, ton i/o
scheduler kai ena swro allous paragontes (ektos an xrhsimopoieis to
O_SYNC).

> De nomizo oti kerdizeis kati me to na kaneis ousiastika to buffering
> monos sou kai na grafeis me write(), ase tin fwrite() na kanei ti
> douleia tis.

To buffering 8a ginei outws h allws apo to leitourgiko (kai gia polu
kalous logous). O profanhs tropos na to apofugeis einai na grafeis
kateu8eian sthn cache (mmap() + msync()/fsync()), opws kanoun oles oi
baseis dedomenwn thn teleutaia dekaetia toulaxiston. To mmap() aplopoiei
dramatika ton kwdika stis perissoteres periptwseis, IMHO.

[snip]
> >>Prota grafoume ton kodika *katharo* kai meta akolouthontas tis
> >>symboules enos profiler kanoume optimizations. Kai bebeos ena kako
> >>data structure tha se xtypisei sto ekatontaplasio sto performance
> >apo>to na kaneis kati ston kodika pou ek protis opseos mporei na
> >fanei pio>omorfo alla pio argo. Opote strive for clean code.
> >
> > ... kai ws edw mallon anoi3es to agaphmeno sou programming/softeng
> > book h apla to apaggeleis apo mnhmhs. Se ka8e periptwsh, auta einai
> > mallon asxeta me to exm ... "8ema mas".
> 
> oxi, milao ek peiras.
> An diafoneis kalos, alla min prospatheis na ypobathmiseis ta legomena
> mou.

Den diafwnw, alla mia pou uposthrizw oti to na xrhsimopoieis fixed-size
structs mporei na odhghsei se pio ka8aro kai pio grhgoro kwdika, auta
pou anafereis den fainetai na exoun amesh sxesh me ta dika mou
legomena. An exoun, kane mou th xarh na mou thn e3hghseis.

> >>Ta perisotera data structures xrisimopoioun pointers, ti simenei oti
> >>den prepei na grafeis ta dedomena ayton ton structures se arxeia
> >>allios eise astoixeiotos programmatistis?
> > 
> > Ta perissotera data structures einai ftiagmena gia na kratoun
> > in-memory objects. Mporw na arxisw na sou aradiazw on-disk
> > structures mexri na kourastw na plhktrologw, alla den blepw to nohma
> > na to kanw otan amfisbhteis to autapodeikto (endeiktika, apo popular
> > source trees, gia na mporeis na ta deis kai monos sou:
> > 
> > freebsd (gia thn akribeia ta phra apo to dragonfly pou einai to mono
> > leitourgiko pou exw proxeiro auth th stigmh):
> > /usr/include/sys/disklabel.h:struct disklabel
> > /usr/include/sys/diskmbr.h:struct dos_partition
> > /usr/src/sys/vfs/ufs/fs.h:struct fs
> > 
> > apo to linux:
> > fs/xfs/xfs_sb.h:struct xfs_sb
> > (snipped superblock & inode structs sto fs/*, ta on-disk inode
> > structs ei8istai na einai embedded sta in memory structs)
> 
> 
> epidi kapoios kapou ekane kati se ena *popoular source tree* simenei
> oti einai kai kali praktiki ?

Oxi. Sou edwsa pointers se popular source trees wste na mh xreiazetai na
psaxneis na breis ton kwdika gia na tous epibebaiwseis. Ta paradeigmata
auta einai sxetika giati anhkoun se projects ta opoia:

- douleuoun se pragmatikes sun8hkes
- einai korufaia apo apopsh performance ston tomea tous
- einai (arketa apo auta) polu kalogrammena

anti8eta, esu nomizw oti 8a duskoleuteis na breis paradeigmata
high-performance io-bound efarmogwn pou na uposthrizoun th 8esh sou.

> kai nai profanos otan grafeis leitorugiko systima den 
> mporeis na exeis ton kodika sou pliros katharo, prepei na kaneis kai
> hacks.

Hacks, sumfwna me esena kai merika softeng books.

Common kai proven practice, sumfwna me olous tous filesystem kai
database authors pou gnwrizw.

(kai o pio ka8aros tropos sumfwna me th dikh mou empeiria, oxi oti
auto einai epixeirhma bebaia :-)

> Milame gia normal programmata tora (user space programs gia na to
> theso kalytera 
> :) Allo to ena allo to allo.

Den nomizw. To filesystem einai apla mia special-purpose bash dedomenwn
(to reiser4 8a einai akoma ligotero special-purpose) kai fusika ta
paradeigmata pou anefera exoun antistoixa se userspace baseis
dedomenwn.

> >>memory dump sto disko einai sigoura special case, san geniko kanona
> >>loipon den kanoume memory dumps ektos apo sygekrimenes periptoseis
> >>opou 1) mas pairnei 2) to programma einai dirty hack kai den exoume
> >>diathesi na to organosoume sosta.
> > 
> > h 3) xreiazetai gia logous efficiency
> >   4) aplopoiei ton kwdika
> 
> to 3 to kalypsame parapano kai diafono opos proanefera, to 4 einai
> valid reason, kai giayto eipa oti an den se endiaferei i sosti
> organosi tou programmatos pou grafeis alla einai kati grigoro mporeis
> na thysiaseis tin katharotita gia na aplopoiiseis ti diadikasia.

Profanws exoume diaforetikous suneirmous otan blepoume ton oro 'clean
code' :-)

Aggelos