From 719e9fa1987570784d7572de5423231a47b25476 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Constantin=20F=C3=BCrst?= Date: Sun, 21 Jan 2024 15:00:17 +0100 Subject: [PATCH] write the section on programming interface of chapter 2 --- thesis/content/20_state.tex | 30 +++++++++++++++++++--------- thesis/images/structo-dmlmemcpy.nsd | 10 ++++++++++ thesis/images/structo-dmlmemcpy.png | Bin 0 -> 18433 bytes thesis/own.bib | 9 ++++++++- 4 files changed, 39 insertions(+), 10 deletions(-) create mode 100644 thesis/images/structo-dmlmemcpy.nsd create mode 100644 thesis/images/structo-dmlmemcpy.png diff --git a/thesis/content/20_state.tex b/thesis/content/20_state.tex index e125adc..48fb42a 100644 --- a/thesis/content/20_state.tex +++ b/thesis/content/20_state.tex @@ -47,7 +47,7 @@ Introduced with the \(4^{th}\) generation of Intel Xeon Scalable Processors, the \subsection{Hardware Architecture} \label{subsection:dsa-hwarch} -\begin{figure}[H] +\begin{figure}[h] \centering \includegraphics[width=0.9\textwidth]{images/dsa-internal-block-diagram.png} \caption{DSA Internal Architecture \cite[Fig. 1 (a)]{intel:analysis}} @@ -69,8 +69,9 @@ An important aspect of modern computer systems is the separation of address spac The completion of a descriptor may be signalled through a completion record and interrupt, if configured so. For this, the \gls{dsa} \enquote{provides two types of interrupt message storage: (1) an MSI-X table, enumerated through the MSI-X capability; and (2) a device-specific Interrupt Message Storage (IMS) table} \cite[Sec. 3.7]{intel:dsaspec}. \par \subsection{Software View} +\label{subsec:state:dsa-software-view} -\begin{figure}[H] +\begin{figure}[h] \centering \includegraphics[width=0.5\textwidth]{images/dsa-software-architecture.png} \caption{DSA Software View \cite[Fig. 1 (b)]{intel:analysis}} @@ -79,19 +80,30 @@ The completion of a descriptor may be signalled through a completion record and Since Linux Kernel 5.10, there exists a driver for the \gls{dsa} which has no counterpart in the Windows OS-Family \cite[Sec. Installation]{intel:dmldoc}, meaning that accessing the \gls{dsa} is not possible to user space applications. To interface with the driver and perform configuration operations, Intel's accel-config \cite{intel:libaccel-config-repo} user space toolset may be used which provides a command-line interface and can read configuration files to set up the device as described previously, this can be seen in the upper block titled \enquote{User space} in Figure \ref{fig:dsa-software-arch}. It interacts with the kernel driver, light green and labled \enquote{IDXD} in Figure \ref{fig:dsa-software-arch}, to achieve this. After successful configuration, each \gls{dsa:wq} is exposed as a character device by \texttt{mmap} of the associated portal \cite[Sec. 3.3]{intel:analysis}. \par -Given the file permissions, it would now be possible for a process to submit work to the \gls{dsa} via either \gls{x86:movdir64b} or \gls{x86:enqcmd} instructions, providing the descriptors by manually configuring them. This, however, is quite cumbersome, which is why Intel's Data Mover Library exists. \par +Given the file permissions, it would now be possible for a process to submit work to the \gls{dsa} via either \gls{x86:movdir64b} or \gls{x86:enqcmd} instructions, providing the descriptors by manually configuring them. This, however, is quite cumbersome, which is why \gls{intel:dml} exists. \par With some limitations, like lacking support for \gls{dsa:dwq} submission, this library presents a high-level interface that takes care of creation and submission of descriptors, some error handling and reporting. Thanks to the high-level-view the code may choose a different execution path at runtime which allows the memory operations to either be executed in hardware or software. The former on an accelerator or the latter using equivalent instructions provided by the library. This makes code based upon it automatically compatible with systems that do not provide hardware support. \cite[Sec. Introduction]{intel:dmldoc} \par \subsection{Programming Interface} -\todo{write this section} +As mentioned in Subsection \ref{subsec:state:dsa-software-view}, \gls{intel:dml} provides a high level interface to interact with the hardware accelerator, namely Intel \gls{dsa}. We choose to use the C++ interface and will now demonstrate its usage by example of a simple memcopy-implementation for the \gls{dsa}. \par -\begin{itemize} - \item choice is intel data mover library - \item two concepts, state-based for c-api and operation-based c++ - \item just explain the basics (no code) and refer to dml documentation -\end{itemize} +\begin{figure}[h] + \centering + \includegraphics[width=0.9\textwidth]{images/structo-dmlmemcpy.png} + \caption{DML Memcpy Implementation} + \label{fig:dml-memcpy} +\end{figure} + +In the function header of Figure \ref{fig:dml-memcpy} we notice two differences, when comparing with standard memcpy. The first is the template parameter named \texttt{path} and the second being the additional parameter \texttt{int node} which we will discuss later. With \texttt{path} the executing device, which can be the CPU or \gls{dsa}, is selected, giving the option between \texttt{dml::software} (CPU), \texttt{dml::hardware} (\gls{dsa}) and \texttt{dml::automatic} where the latter dynamically selects the device at runtime, preferring \gls{dsa} over CPU execution. \cite[Sec. Quick Start]{intel:dmldoc} \par + +Choosing the engine which carries out the copy might be advantageous for performance, as we can see in Subsection \ref{sec:perf:datacopy}. With the engine directly tied to the CPU node, as observed in Subsection \ref{subsection:dsa-hwarch}, the CPU Node ID is equivalent to the ID of the \gls{dsa}. As the library has limited NUMA support and therefore only utilizes the \gls{dsa} device on the node which the current thread is assigned to, we must assign the currently running thread to the node in which the desired \gls{dsa} resides. This is the reason for adding the parameter \texttt{int node}, which is used in the first step of Figure \ref{fig:dml-memcpy}, where we manually set the node assignment according to it, using \texttt{numa_run_on_node(node)} for which more information may be obtained in the respective manpage of libnuma \cite{man-libnuma}. \par + +\gls{intel:dml} operates on so-called data views which we must create from the given pointers and size in order to indicate data locations to the library. This is done using \texttt{dml::make_view(uint8_t* ptr, size_t size)} with which we create views for both source and destination, labled \texttt{src_view} and \texttt{dst_view} in Figure \ref{fig:dml-memcpy}. \cite[Sec. High-level C++ API, Make view]{intel:dmldoc} \par + +For submission, we chose to use the asynchronous style of a single descriptor in Figure \ref{fig:dml-memcpy}. This uses the function \texttt{dml::submit}, which takes an operation and operation specific parameters in and returns a handler to the submitted task which can later be queried for completion of the operation. Passing the source and destination views, together with the operation \texttt{dml::mem_copy}, we again notice one thing sticking out of the call. This is the addition to the operation specifier \texttt{.block\_on\_fault()} which submits the operation, so that it will handle a page fault by coordinating with the operating system. This only works if the device is configured to accept this flag. \cite[Sec. High-level C++ API, How to Use the Library]{intel:dmldoc} \cite[Sec. High-level C++ API, Page Fault handling]{intel:dmldoc} \par + +After submission, we poll for the task completion with \texttt{handler.get()} in Figure \ref{fig:dml-memcpy} and check whether the operation completed successfully. \par \section{System Setup and Configuration} \label{sec:state:setup-and-config} diff --git a/thesis/images/structo-dmlmemcpy.nsd b/thesis/images/structo-dmlmemcpy.nsd new file mode 100644 index 0000000..86d61b0 --- /dev/null +++ b/thesis/images/structo-dmlmemcpy.nsd @@ -0,0 +1,10 @@ + + + + + + + + + + \ No newline at end of file diff --git a/thesis/images/structo-dmlmemcpy.png b/thesis/images/structo-dmlmemcpy.png new file mode 100644 index 0000000000000000000000000000000000000000..dea8188cf782fa769a9092d3becfdfcbdd5f1ba2 GIT binary patch literal 18433 zcmc({by(Ej`tMDOf(!y8(n^aGBHakmEhUXKNXO71sECwwhXMjacQ=A`cQb^<(4A+E z-|yc0x36=a=bZCA`?`Mo$z1c9HEZ4Lj@SLZCs0X25*M2k8wCXgS6WJ31qB5Sf`WpI zfBP2r58*KD1`3MgFKKa6RaZU4ELPNgwJB~pX*9pv5soS%A_meypIg4i+3AbZGVVTp zs7_a%E2*!mY(Gl;9-aD;?88q*5wy6nwSFlsE`A@I5R00cdOtW29UmXx zt-bC2=g*%7jFv>eYbnp|AA#5U<9P1j;e#&CL|`zZJm`waHP*{34gkr-nN&x|bd%6v2?<`f`7z zo8f3*wm_#kL#Ntd|7cu4cX6@6icNz>KgK#;qLbUC>(_7oov7A!_W&))+h)m5Zi+%@ z8BBN=84@9+mV$=k-1EPFP20xWhFnGQ7=5a~3)U{2DZ9i5-PWnT*si5hOc7vzH^CMd z6!iK%9(3U67otdf8j0*j34$IjlXY&TB=-l)%?Dp5aNF7()0|e?%xZbrjOE2M&GUKH zx*(S0%~F(8r^_q|+2#^>4!38u`J8?~()T{goN??IjG&W!Fj--(J+Vo;&_y&UXb~X? zQ%Y2#-wF+ZiLpiS*j;a-s`Apr+PIh8fW@|bbY4GEf8yi?T zslZvOTNMAAZ$s^hVJ1@=6>{4*E@vBI(y6i&1>2T+esQ>Nbn7R3I|YA|pojVqjVnC^ zL$?V$lv4CQbR{jTai`%z^@fPXmzY&gXPaaDI3J!&skRVG33q0G$+Kx${!L$)#O7|Q zUS)v}kIORyooweaoN|SlIs!TwDVZtwXKW&-Y*PH%*Zk0~5Bn>lHO0r^He5y-2DYY& z`X9ES)ojn!CW_)ik4a%0R--VJV`644RhgJ)IYOK3R9=V5j-Fm{2wCgB--tl;h%o4! zo21|a5q3X=pglqc1yy+K8;z&5IJxsa`tA1_AHjoY>BHj&NxZ<`SjFqtuqc^#EF{to!3ESGdi(t*5Zw}Mf34s4STTV^nmI23e?q-L z$HFBrFmN2hT|}fzy%kHAT&W(f3cbS!Q{bdBbbRG!TU)lb0Q*FZbME2B=+j410i^}a zk9cg!*+k0iAI8%JkZTq_TITyyt8i?MX0=pg-FBNKg)T@gM_t~ftzD)oQ+Cw}9E~@) zWvI9~%G*y1>!_)GE{l`+oMo_yS@LT7g>Q6jLC6_(Yn+rtUuY^FM)0JG!8@~3y|ug~ zJWw}Xb>{}YiK*Qj^b51TFzbtgspa0nShpJ+m6vfS*&^pwfUxRQI#9IgGF))I=d_pN z=$hd6sIY{Jii=0-4ZTgg32ePQI~aX&n#Ah}Yq|seU@(eCwqJ?dg&glDlpy3h_c$2c zrOA~ZP-3Pf+!S|EM$$-zVg!$`Bnx?I$p<*+Xcm7DrWDkQF7$We+1yaecD@Tu-kQcZ zmLx*@2WH?O4m122y=zEvY(6>DQd47(%O~^jTv<{R%-r3S0$!4Z9W{B9f&8yLVGK*4m^**=USLITZHzcS& z@A942hlZ*h%`UR#!c#UQQ%vTwwU25g#u;D)Sas8nt^G$XsCHv|iJq!YFyJJzocLU? z`JjD*_r|-sb;mH{JS5Jsd*4J=HdMF^p&#*vjIYnuuFN$1msqu6oc(zB=7`kc(z#cu zr8>@fzCpL)bP4BTK@K{kS@?l%{He;Zj|7j;b)CD~rCF+%DV1|;!CI13|50^HNue28 zfh5m-`OW2;utROQugz3hr_dK0{PhH$1=*!6C1OiHx3x^4REm)o^F zTalk$F+fhAM!1SbACJ6NoOKFBp9^_k`E@BIUE$EX{rJpqv~~x^?Y#6;`?=fxuZJ`_ zVvScJ-(7&`KJa`FJfwD$Y%?{co}&;~&(W8tE1%M!5l3orl5*Td&+#;^a?8+h!!O>* zRTR=M_0!QTEGe>5@j3lUGHv~YJCq?!c$BcN(vF#LJKylQ!%dp}Fg5d2rfe*1-9DoA zXAtoyKZ6q4d*_yErtw1k!gfOv*ViV%;f~<^$b&93R+{!Ez@A+`F>lIk438Tr(B*$* zuUYS2?%t1a-R2a{q}DTTyy@JLTT`sRTaxR3w9%<3e3fyG{}1o}A~tB;4X&tbl)TGO zDSWm}SejHh=PpYpdF+)t{`$ski%vEs+q;eselqW)0ma^FylKcXd9>HcUSz{*IifWE zgBQUnxKm`gT9%iz?oB&0Xg+N?hcI*CkpfLDNGHAfjB45p}Zc z6?Zm6)*CK1^+%^A_t=Zmg>UlOjrhGe#&A%1+cgVG+TD}c#&E)JNj%*A=r-7kU7{CW zc9VQ;Tg1LM7h|xwO{+S2&L6T9AQW0d#_Nzlx^_5=O+UgLKDdc~T;eQzJgpycfZahd z4oe;G%T=1bXX$NM6XQ1x#ht0ZTQTm8cy+upZwpD+mYm{^7Yan@n_1i-7AUtEmI#p? zVAiXvI&X17OpV@cNS6q4pfI~xnw%uW5@Bl>aFuKWiOZ;&PVmLiW)@2igF=$h@DT_^ z^}ZY3tc|L+Wo@x?xc#LRpEC(%&w6#^eRfV#}|Kdjy%BCzU15rnlO9rkc9`t#AZ*=m}pUN z?@E!zkWAP8V)Upj?saWlcZHu)@3L~o2=?;q+%v8uSxZ(RALxA%Q{6T}bt-^R^ieU{ z28VWq6MWQ?+w;5^)2RNKWHU43j!}KZ!_D&X3cc`(zs~SvYm(4uHu(dSwolgH1Fbq$ z2t|YB?IfumvkltKANUy9-MyOhDvo%J&dzymo*?S5%W=Q0LTh6_ofmwZYT&dQ%Unz5ez?Bjn6Fi?de7NUj%s-QF4@X=FZ69CJ1AC_|II<_%{zslHpBhp&W#t_D!WqX zP~+b_!&za~+pcmbrz+`G#GjUzjgNk#5Z~5qVA-rcnUmGc;W+m4Sm`2iwja1SKHY;y z&`3g#USZWS;%xe?57CjZ8Tcj;k4D_HB%zauQtE&xL0CANHBVwwD!<|<m5vH9r??Y>2C-5T4<6%Nd$AdiTQbl4+WYfpm@HQ=v5fP9NDC z#W_6qK!E|0{c!*nhk^70y)z(TT$438)iN*ET`UK4>OPMk>~q=GPMME?^m^I0!pmsu z8->lL>wcf>*GIKpl}S~}gz?2f$XjEHwkvpx+zi99*P85Q?~V&I?vXA|xz}1>GR_2* zlT~wZBks+qbn1ZXMfxX8x4W&ko1O zWn3Rb`S^%kT?*M1Bl=6`)|!GbZu=TUp~2=_w3kJVs9f)U15QUABk%*~BuZslb3?tBDZ(e~iY!kq z%Na@ld$$QaUugD+os);ne%%E=r8{t%H=JbCh?RfD{#K>+ZO+E&@)7ULt$9w?>423q z+*`BmLwy0jO6~m6wtwJ&R3)XkxpLk+xBX;eRQ8BU$W!y^W`)(q{%=PU5dXpqw!Y^% zfo<~Lk_!ImF3I->fzx1^k3j9ul0jQV^1dYA^pu^Ypf7vbq!siKzM!4K^Z?k>2!+-& zENy6|MgUy-_K!-_!>Wt29c|sj^b_VZ8N%U(x7p*khsik&R(H$qZq{hAO4(dIov71W zzslPA>asJJe~!Sd*y_MjdhH+N!RQfeEn%l`J-Pa{oonZLAHN#>tIOP#;WIHhoIjgY z9b=6xMLuMZ^0D3BaH`j0le1Z~P&44@OW9T732)@qF&c`{s*1djkF?gi4po|FDlNkj1o0S zr}L(G9yoD$~k70U#zG84~LmNHfz!7 z!R&Fmy9lG=jHtjge5T^@$feErkkUM7CBnblY8y-!&gM z83~}#E4X5)CoN3ny(!J`Jlo&c{w5Vp&D1RNRnY6a4EE_~AuW@7UN+04(|zDso%m5X zzF=4_p=d%!q!zkiWD$$RkU!gtg5l~pSxhw$+MFO@NTO+`Q$^*m_1yV!QOhgx;j5jS z>q6eU#4iL|njB?&tw|J!X)CXY-p$#i-4!VPDJ0qYRCRRY&#qohVXk41-RVF&BTK714LN!x9lx)C z=OH{nh*fF5%icI#et)LH%6-mY8mnr1k#AGI_2uAGMD2p@lE}KG{xhZYrCU(nxV~#i zEhXll4aLev_&r=FkEqjA*9%4&$(fzXcA=(1HoMlBmGtxw?sx+ziMly+D3j(4E=Hp1 z&RqQl1YNCY@GkEB09y>Zy2Q>&Mh5NC#K6?I2edCt4uyW1o6q~{+gD#5MtvtE3`msA zZbS9qBn)HS#WLJA_XAKw@xOzOlsZY14`G`iVGf`=uSC|e%Gj@1qX48~Pn~}rk1SG@ z?;r5fzd%x)FK@j&jQK!O$I9Go-TMr3 zagUg_E6VWbsi}$oEGonG&ddPFxJkI)#d+=G9@JcdWWN(og)RTU&%fT!Kj$4L3_!!g zmXebTj$+Z(^72Loo!wHiewaf`llmQ8zh@s_Kj4wysUY7< z2e{)`+;gLms9KbEAptyki-5Kw}f`hbNR|N8nk#gB7auKfF}pe{hZdan~Wt!^D!9jx|722i^H z65>JW(yMbVp4>43n=8g9^%jYC*zz3L`i5$uwnw~4B~2FYl`Cx>9kTK4CTQ)W1-jap zHX|5mV-m75G7-QF!ZJ|G`50S~OOfKUduAsGEVs<=Z3!`cI1jm?`^eVK^!znFIu_(E zX&#bL-OCH3;5Q9z)%^{s?Vf9poyYaJVBt|d11YJ>=)9}(-hw|~!Z4=-ug=Oq(ham-jv zmF19rT3chS+kH6^i2k9J#ip6E{DGA5KoMd8$#~uy8cduAQM0ws+`YvpOoicEW~{(_ zq|r-jAsPib%3&r05*Y8v4jyOYQShmHNtAqWd(6VJo!}JV#|U{d-&hQRXZY`jij@!=f`dh+^*RL}$FKX3Rr{~Y-S;6QCK|1;h)OM9v5FZWy=6x^q1VREj| z2r#>e(Em?=`+w!U&thcB6_RBlhB6z&`;vXi!oD!}CUVE_!8@k00-~6;xp=)VtGw)L zno1N>1T3hwg1Tc_k(Y6~Eszkn>`e1%O6nT7Q;JC2>bAwpGh?^?PPhF@v&IR@S5ex6 z_P!RwKcr=3zIg3AOUubc0gl~W?YOp;&fXL9uK&Xw!VHt{nB!(~@y8$}AfI5`+}5m* zm@1D1K64UlUe$6G2H426Vh|4;R-kfyBW;UIKf8xAyTcq#W{s9Z`QK)EUy+n}U)`a3 zNJnCFmz(8K8$goSE<|NmRKB)i7feku)37x@;^{c!6D_AC&;3q5kMORm6QA5wUN1iB zqe49|x#*J?9N{<<;pe>2>2J@wr?Wu(U^V{vFgoC zXh6GtQPGXtKI5yvRG*CAtY@Vrto&|UTpr6uf}A%8%~+_Zt<1KEzX#iea=w2W&(%-_ zk=rn|lpbPRe-tBdu~oVe>JI9&_ToHN;GDTbHN+;%tzvn@bV1BA*`CSmcnfw;rAJdk z6`>vZD2ttp+vXYQh^VKs>eZPiNkV_c>U&$n1e5?M`~>jlEPJ6O_11`ph~>;!Jry|M z+q3{%mAxTU%8aa(FX=pjZO+e+`N)N(P03g>Sf-Ck@9#4)vN z+$_)}$_szCHR&ip_fU#d&&MksUx%1P)Vj7_DpN7_$nrXn5Sn{^GOwS1zScJjm0$9` z_QUKTB|hJi*qx!5Pwb1?nw0EX?QJAOJtXEt;0cqnqfOZ#Rog8V@1L!0t_)?~?~G)S zpd)tifG-)V_9ZzOG^1T)r`_9?iDG0-)|#S%=UhVL-3y0aW(rn%r} zP;%boa7oJWdXEzrqM76NHVAI2SB!)goq^hu&6fUOGXzg&95+g*UC_Fi-jrng*P|HD zVt^UwR_F`{Mg=gePuRU)ERk_ZMaik*_G6HXSFlJq)LV*&2sb!(*grS zmQEzkZwKLX^L)^1c{6I6M%CHTFP!DQJ!%z#&%-4X&p!W9)FzO_R!Dyd9+qu?TtRRV zT)P%tInOe@F%ph@b&iQ(xjxoz64=|@gDobh0V72xz?uoPd=>N{dZjxy@d~gbj*I-} zCv;EB+kam3Chh0hWnB)4;l*DuVnHv=`O{(p69_T1aG?utN=@tO4J_-7Zrq ze*4O(@g`4b)7t*&K?jwO>L!j9&nHl7l5e)O7uP)kq)?@tJ6m~urn;0@7>%vMZDS;7 zyo|Sr9VAchj*)?LRLGSG-~;c82CrJ!5w9mt`K@kie$+cS4ccb?KfD<}f3DjljjZni zuBVGgIS|DX|K0~O&M&Xu>tdZqQY9NHT#RZYjugekEtKDN2_fr`@H~5~){DBjImXO+ zZ;BA)cq-lH(AnCK+z!ICgASQeQ^!pC#E(q)XXl&Duun@ttOO%Yu04iCwitzr?Z@fu zkLeXt(q^B?It%|vGHM+*)3GsgqB~dTHkw=$DSUI*zhRF_`FbH#dw6sy!VnHB;;+cts}x`jKjZT_E4;S=9XD z{SHO5;%efP9zay zV0?Ql>sKQdrOVMJu1)_GNKqgq;g`Bl|(-Y7YFa*2?sAEd(6 zZ_LMS!m)puR~q1L+igu?s`>lg3bw5)mYb@)5xw3Ei;fvW`xczUE1APZK=f4M?n4`9 zGM{nOu;)(Gm;w8=7ZEnI#km4C`lGxSYnLjy)o(|yh}hR01<%?fTAp}^~n7b(?O)7P&jZ^Imh_E zw?1Z53?fjMmX^@Q$igHclmgj8>7D{pMV0u2Mb9{##M3(9_j|Gk-|PEasg-*a?W=IL z!(ENM0s9Y>h@a>e7hnAQY8j5@)wh3i@H;no8BPsXP46&3+~eR-$-$-9A#ZuHfUwx0B(|9sp43A0>k_-)p;bJThKYYmL zd8%F-Oa>bLGtc1~M5pVINLlBkmKKh4u?trg`znAl*>6s!6Up+)U#dvXmTs;V z6~7NDd_I{A?Bg@L4Cpd=UGYQA!`|!L8Pjsay~>B+EAxOd8??l}MnzFaKTTJ=8HI2X z+;`jcq~ydt?mOouA0nj0IX#P|0WAru_36q&kf(vv>`R&XV2e=PXGYpaqkYh-MCvp6 z8;u;q=^&?c=u?p2jg4PZPD`HNXioE;GNMJ5%@0n=T7Gt?%i(yb@pk`M3AFT1z>`2kY!>dBRGn??Pnh!w){RAvKAo*H)10&zrdaev?y(|8?X*Ei@G(Mb2>{Ta>0# zeTOM7pe0rptXYAz7Q4KW^P(H3M7BU@Kd^P6eZv1|#l3&6IDkg7FLw%qgUj=vmq?{@ z`_Q+od7Vx(ZJv0q>#1xAe`opFTjWAKC!>eE3ucZhr@LWA-ZhWuk(+qGv##B^ke$;W zg+u3=+J-P}%4eo3c?19NuYRz$6N^nm4-*NEeNzpN-tPw1V&yh?Uh>p#YTkx5H-W<< z!csL5V)FKCtMQ?IqP{I2a%m)wk+t6s28cLGAZso4ckK`WY?uhJ8n!l8n`3on!6|?F z+yXND&@X?)MSlMoB*i;Rfc3YSN!QrPTjuHT&C5Vg7ajb^pw|CMkrki7lOzytEXS`y z>El`86}3Y^*q?1uZe1&NWVT#VZ#h?29ZV%WC&ZHhm`^7txe)%gaa1dM4W4hhqMp!z zE@n|!7n}DxNw#p8y6+#SukFd!Fjs}$_A|gm=Xd)3RbaOXT_sHf_0QceGvltg3SSt> zp?#uzPPRUCCV7$VWEO_L{^UjFa38RcFM(pWo=xXU;fQ-j^IK1GNJ~pArxHKYT#${o z+3K#ELjZ*KuL>;GLi%;CgIr_$o5$1rj+tFED=e*JKhgILH=x>j3IJCH)0TrY-;I?siP);WSYPfGRGsizZ?h%Y0l&FGy#j%QD&vbZcFZ zN~H^f*BKOr=8Gif@J z;T=7;g+itZ!OLKXw1?Y%P7Y?se8R6L4QNAffA8OD)8U=P{_pCGGh-Q*D$X6WwX) zy;;`!$vQk%UyrGBUPz_Frv7-z(Q}rsRj!S zQY@1uUsU$DZMUDO!J;;94u=(c9S(mk|67e1`B%Xf@m9j;+slv3ppI1ML^11Ml66X? zWnf4&F%*;tCAWwHS*d=Gb5q^T`FLZ`dV$%Mp4XAuQMGs=LA(8vTuP%RBKXa}smFj~ z;nTeHxGu74iBT>yS3Wz1hxM{chEZCRr!!f;`5TO8FE+<~s}w7o-Yje^_%VCb%gw%j zJiRi#F-m9Oez)}9>D9^nih_|Yl5s-1v*AQGZi!sh5$8vnWNU8T786C^wx-Iv$%z$s zn`uFxgWYkZ%f$1htFt|XY;-rj{hZ5UNSwJ8v@6@8CNSQxVf$R&+KQf{t+k`gaWdjn zKL^mab$nQr3>|A)uYf3QTFaBqxomFoJqpPVJ~$(+hm0$ju%J7! ztoq?~Zj!lf04o=f-b@%dj-EQM6>PKeSMn^BU0q$+vK{Z-=47sE4>6mRk&h;sUSFx2 zatrCydnLY<2DQmjkR4g$bN&%Oe7Ie&z~`j-45Yr&+tp^@jXT+Aew8TR>IrHS>%Yhn z&g}C5;m+C4&JT*H(V-(6BW(9p8VuVi(JbC;|LB~^Va=yqVn+2Ok^9C*!DC#vc*A!o z$Z@F+(~-i^RwpUAEB-a)ndUvVnLDtjdzn-(OoMCedQ*kxCn}zqK5BV|N=&1ps*$f{ zk|*n8BX$z%tFlXCV~I%*##!07nMD&){JTt5d%z+Eh_KRuTeVdE z<$gEQ9*^CEuWGiE=xU$*tp0eZw^E65XD(D)qu|%Jbfj^Fy>cigOP`hdQAAMVRSX}C zwNU*fvaqcnC^gOWeqSt?M+kU`7@y0Qc^`4nU1{blzzrQ1TQJKhCm^Z^Et6Llqe3kW z35FQ@y3^0i`pAh~%Yp^;7NY{%ZcqD*;8tZXys23&PcmS`BM1g;oRqliz|y@Z;9vz= z@2uT9{t0lR8j5%tW)}RD@j~@l%8PEFo%$2MTQf>)6vxTad_4J;DO7(-`wF+e+(O@%(ehJiLW5@ z)$wA{am|xKVbPU&PEhiY$SL{57j%W`u%ot$@1C@Z_VymubXi@&h(iY*1TuC5m&U04 zwr+2w?)xXTY&K8|WoHVgn@>=XL#wdd@Yowl^M+;TD$j z{<0wKCOvG7l-R$UsZ>H(;jW(IHeP2IwN*#eEA9#lLLkN1AV0T9t#QS z=_=*hNt)t{S4DysD?HB}N*jSj# zDa^59!?W2xJgPwQl6&+FS-=~-!etja*(h(9!r&&IZ?;|jNgOM6uzC=(1GFxE`M9Fl zNMGQq^+hR+iQS05o?L_f@5uPo%_7@cgb3eYt?{6*e?u9wM40w|9Oa~Lwp-)5Wh5yT zVPpb6T;z@7!wXRg4n}8pkMVksI$lJ2Sf8}+SNS^o4qsn?(ttPtH$RuGAs98M>ZI1r z-Da($5S+H>vTWs8t$Y#@-%}mmu+l5sBTwM6hQVZxbUliKRDqL5V^Imf%TkYS-d$`R z#Z(HbO*tx62rxE0y4dNCL&G#q!XFrWh`JhfbFD%AZI$0)`SWBL6SV%a*&j<5fEHCE zL6|3%=dP$D-$hMYQo3ZJP|M4EEwhY|44d|XIMJqjf>DBvS0jq`mFnjJS#V4V7SaQ! z*E&qj>yZ6H1nPFMss^200M}Ki!YdJ#Q}cKcJnsGEX+NTBaS#Qzj?|?@f7Ts*)A%a; z0CLf1O_4N4#^&QjBU*=4_gcRvtlV!#Bw@Fw@bDxz>NP|*H^?3zsC8TDL?&5Aq0~}| z5^?KPwR2kD&nkd|<2$ypWXzuHpfR3NHWDf$@wF?c&#vg8`DEI=N_t`Vt;QQ}&xyuI z%JP4{Pu=gv>^Og;4M;5Zv#N{MkAxjvQ(d({W{yhyWK)-jMd6rVxOek>aeIP~ z;*pXq$9CL6yI}@C#u1#dCtz)X6jo7uIaz9!iy?&-3)CyOXd%ZY6XR<7%atJyAGeko zhd&vGb%su8N*4=Q?aVhe)&z~PpS!v-<3Yz=ueaN*c$*h=iszltE%mpchpY{!-|g2% zB)wetZ0->gGbjq|ec038&h)lOo2(QZaa%(MW865Z>9CmfX0-fOb)kt9E1p3A>Hl(# z$`u(md*WZ9w)*pWp5hS|*(H;)8#@kI%|$)Qf%X7Nsik9F!4t@>(Y?3^qmG{&K01oc zdKJ^<#3^p89!I4TGrjQe@`5hOugpp)VUXnU$ExH5Thn;vuLBGyWh89PcskS$0IK`) zyUp?2fpr0%atBWgB#da@Vat zMbKe)L6npE$5WlEqWJvpGaoJbkCZb*YBe@rVOyiy;kRD8h6b7CjG0*Kjo}Sc-BbJ7 z{K)l>s#zO$xCatt%bV*f%X;PfAPI|XK~w>_&*Dd7Gy<#(_N`4A>waK6rU%dia;G}K zdDQf#yXP0&Bw50d2@m>1${bKmm=Ihu|4pQOI0o=~{v`uE_?)-QxqTf`?l8poHw7%Z z9)4W;77pK@&FObb4ka_t!#LWIh{&gs#F#!-TKh+IvnE%<@t=x+6@-TV`5(t7mVLlI z0%f0oc6v9U9kW&_Ux2ZLLW|+&mLpuFr})hT+~!q%-)3JHSHFK8WB!`5{JjTlFNE ztn#&*ATJ%{>sPy%g6ATeTC%ne zJDm^s#vB3_O$T}cg_6;XEEDI{A1(Gshz*mn&(BzN`p=s)q$5>n#GJmR3>pflu5XMT zrF!lMl7`CFB4LpN*=W7_L-Ts`saQk(aV286rbF{go8FBV8?JA%24}j4^KXUOL;{&dRE(i;zI3iXdz4qWco)}i+B`YT|a<2}&<}R0-{HloD_X>6$9H$^=|+waT0jF>Pi7-u$Q7 zR%$DmA2s1n^{TmA`DOmz?8m9|$NFL6_jhBb?Zu`sFLg1jT3dRS*D5aFj2#uF6u$l; z_na)MWnACKOU!3uN?Jwbi)Eb+LWw4PaLDSSOlnf$!v3i1k4l9)MjPe(uN4s7144IC zTXI5h@H6rLGtJXsW_yEwXlEq;yR#5vc& zI$ZepvpN70R=!&*^16^dTMEMHR^_IjU>}mrpoic*c=i?E5teub20&D*9ZYzk+evb9 zY@%TB$Ra)gD1rR`>S@NBTbXbOv*ZLT?y+t~Q~T zbC#n&e)S7eajyLoV$?n+>r53^7rOEE`qkEKLB)q~%mw`?son(6OuNOFp&#;OZbzdU zCYPssmNP%(6N9hx<~74Ehl&jUbX!S7ElNzf>499BZLBm8=2msk$G>v8K5SYIwkqNn zF`WpcFj3ck=i^nN&Dogp2GioomeF%Ov1)mA^op9 zF&gF3;_oTvY8j>(0<1%RM61CC`Gu0KfdU(!)T86>`+`oge_&Ye@LdW+VAGVlRgn4Ux+ z0X)Rl;}veMlHqsk)8@1>^x<@Rk$!F1h|jXjMxLb}E%v`xG8$J3AlDGOThX5n+S@sn zck@7Hmqic&TJjG!HM_`DqM~Sl3+J5X)Fcc*MZYBlB#XTkKui#{F~j3=JheiAuY8Vd zMtu3)wm2xc7T09s%iy%cE~ihlC8<5Z)aP{mkvixHtri zsuSfL2yiOB9==P}?u~m+llA}=Ideb|fK*I?CU1uM;CItlZd<*{YR9aEz674Xl|_nH z&t(*hQYpb~3t;@8hXRppcA&Z%^DWpZ@HqU;z3m;AK;bQ&_E_0`*z{Y1&9AS$FxV{V z-N6TF?lei(T2nI?@^aqHh#>lq?h=I&l_}WU%})>3-zp2 z|4L(bBJqTE($AqgLiv9!**sNUR$4**rlSM&H1XB(gfEL^$bo{N9!$buKQ#L|bvv2o zaA#h>#%Y~^jgs>|4|iWuwfnQc&x~GwK!?tf-G*w!eHHsC{f=Tp_ zWNbm$>GuQ9>$5eVukkMkK^z)!!PX=A1H`o+lu5h(1;m{=>t8P|RNx_oTLM2|1a>BC zFV#XdnkhK3K-5u&f0as$K@CKo5{_ckpPPvJw(){eFzs=sdb$K|yntKDIXOss@q|v( zcp+RiGuf6;00V}*BFT-;63U%B465aV-n`$)b}4x56ij=ktS}1hY{a5~&>T6gum6J* z6@`@NEjj7cV;&l%8afC7>KcgKU*DCzb34NCIKr;DYqcv36e**h4cuIV{XSs9>u4dq z>jSsY!Zg}LrgJFEdFUVuWqAJb9uAx8B>H)Ad1njgqhT%(pek&<&!{8ihAW-dD`~M_R{Xb#(|4X&&=X=QTd|_QB0Hi|~+1h$Q*@z4r$IXv{u1P+W zO4x?VQ?`1}YYp>w2dGgH{(+1wNUc;u+u{IAc)b6Xn5{c-D2QprE@FLJ%^&-wU<^40 z-IQn8G-eamn>&K%L#cjCJXgBzPQ!s;Ao^f$^G=-S0UhM|&S$3_b1L^?oERRb%?cgt z2jo`qU5Wps=>FJdQ#}s#%{vkV0d~i-;CW@6vrs)Y`$sjrU;!_z*j*a-JAzp!P~3KJzgVzuxox)mN_3EtQUbUS#<1I;@S+j$q&CVh(Ge;Y@1u z-6J`Tuyr6y0{YrZM(DgC61hrjjHf={98dk)8gQeuQf9ta|HfsdD-uj6$N2)K)Hk0S z-1d)dA4oqH*u%@bzS2K=BCY4XPWMDTFYZ2*s#J>ixdmuxgd>}Tx&yL6J z{`e{){9=IV@nca^u6K+;K^5V!@=w;y#Q|71nAc?mev`egK4(?H=E4Rk@H*4f?+tK( zBBnAw@gq`|x=a2g0V|jm6OV6olsjf*CkRk5dBqn)aJFC{P7|2JiCw7r;EnN&^@+{n zp@O1c-tf@vSWN2Vry2B^QW#@B2Pr%gvEL%Y`n7}Zy3rgR@P`z}giBx@)^2}^2E>|- zcqYEYte-@UP$_T1mB!pU@`t>c?C}3#!Xv=^!WGu;&{a!2aA z-3@BNB3UT-{sz)gW7}WBpnUu=wLOm^#)7iMuI?Nd6*a50?#lDZHYBAL1M;Y-1Bs!J z)~#G`8<4u^=zyzI_an>bo<~i!2!2{&A|}8U1fGfArXqzY|@ zEPTxt8i#SJnfA|_#Sq#op@8-h{4^U)Gz%mG8;}9-STj6#^?yjGw@ye8XJWFhOlJnS zIsBsP`D>wtn>@K-r~znJ<(i5>F@{A8sd5}EQUEuA`@(80&*E5{N=?&>>+tW*0g4Ve znOGKUsyu>+Bkweww`bnCpKK#)c9aO;i;IgPB!K`#_XO+&$^{e*-L<>voYhd>0a?-k zd8FW%;I_p`vjRxs$xd!L>X^-8Z4SJi7BXw8pU9&b`z_!bmzc*1YfhNxJ0}sJ!$IEF zB2j;?9K&NULFO7A0Q5FYozYfNq+{92NlY}FOG?E{ZJP?@D5xbM`;p)!G8xOu4P)OD zwcTB4Zg=NU86C;C@c|NSe+9Bn1Xg%DXZ$a3tWAc?ZG6g$wj95Hj+i4Q4q)5 z*O2cFglEW^+J3z$9vh1yNiCbTc==m(>Hgmg>^?+B&xb zSi)Tfn{KqNd7xGA24ezdv;)LwKWM)HTI8koxK9$;nw4Jw2}(_gVZ1E-)yJt(Yh(f4 zu^qYZRy;Gh-hcTZ?qOOCnl;TVv<9Si~X_}fq|fD+G166AQ@T}TKdx96=l$|IZ%@b|f9Wh)u0(#-Ah3O&R_P46seioYDEJB9K(2O*ElndhN*%A}k zb#OnMoW$?*3UnBUQM1~g!pnZip8Rkb#`@fIrnS6jlM_OZ8bbfPztQ7|P#=oGrzWFM zRfXcWoD?WhLjzlLUQ4S~?P`vCU7xv=@YtnW-V2n7OqZ~J9l{v?tjwxXsuxVWx^Pla zLYI4ZWzLV~*&?yQZwQ32KHrMgi0SH}Dpc!FHF8*sIo+N$`2*SvHqwbPI>1;&V*l|V z4@B+PJwn3tZ9CNdYJJzmkG^SU$q(**CMHFt*p)(o=G=C-={_JtVp7Uc1fT6Oi%t_W(!) zyE^I?%6w*GFl&HT%2&z17-*$7>F+6j*hOB3x;E5f8yTQnosUu_&(UF@~>M<@lf3rP3Hy{ z1P0t;;pBWXblg^BadP9`MvP)0 zvA;EDhmWtz`kw(b{~S$n0J-$vKU-7#>-S}p`j5ts1Yo$y>Fs~x&(wessy}mA;Ma8e eWKV8T?@%b0KodVNgP-<6k$$BhUhvYu?|%T!1EYul literal 0 HcmV?d00001 diff --git a/thesis/own.bib b/thesis/own.bib index ff826c3..bc29b6b 100644 --- a/thesis/own.bib +++ b/thesis/own.bib @@ -169,4 +169,11 @@ number={}, pages={1-4}, doi={10.1109/IMW.2017.7939084} -} \ No newline at end of file +} + +@misc{man-libnuma, + author = {Debian}, + title = {{Debian manpage 3 for libnuma-dev}}, + urldate = {2024-01-21}, + url = {https://manpages.debian.org/bookworm/libnuma-dev/numa.3.en.html} +}