Intel compilers. Why were new compilers needed?

Intel C++ and Fortran compilers and MKL library

Along with the standard GNU compilers for Linux, Intel C++ and Fortran compilers are installed on the clusters of the NIVC computing complex. Currently (beginning of 2006), compilers version 9.1 are installed on all clusters. This page is dedicated to describing the most important options and settings of these compilers, as well as their main differences from the GNU compilers. The page is aimed mainly at users of MSU Research Computing Center clusters, but may also be useful to other Russian-speaking users. Issues related to compilation for the IA-64 platform are not addressed here.

Also, the Intel library is installed on all clusters Kernel Math Library(MKL) version 8.0.2. The library is located in the /usr/mkl directory. Please note that subdirectories 32, 64 and em64t are available in the lib directory. On the Ant cluster you need to use the libraries from the em64t subdirectory, and on other clusters - from the 32 subdirectory. All necessary documentation and examples can be obtained from the /usr/mkl/doc directory.

Why were new compilers needed?

The need for new compilers arose mainly to a) support programming in Fortran 90, and also b) for more powerful optimization of Fortran programs than is provided by the g77 compiler, which uses translation to C and then compilation using gcc.

PGI compilers (Portland Group) also meet these requirements, but the developer company refused to supply them to Russia.

How to use?

Intel compilers are invoked using commands icc(C or C++), icpc(C++) and ifort(Fortran 77/90). The mpicc, mpiCC, and mpif77 commands for compiling and assembling MPI programs are also configured to use Intel compilers.

It is also possible to use GNU compilers using the mpigcc, mpig++ and mpig77 commands (Fortran 90 is not supported).

Input files

By default, files with the extension .cpp And .cxx are considered source texts in the C++ language, files with the extension .c- C source code, and the icpc compiler also compiles .c files as C++ source code.

Files with extensions .f, .ftn And .for are recognized as source texts in the Fotran language, with a fixed form of notation, and the files .fpp And .F additionally passed through the Fortran language preprocessor. Files with the extension .f90 are considered Fortran 90/95 source texts with free form notation. You can explicitly specify a fixed or free form of notation for Fortran programs using the options -FI And -FR respectively.

Files with the extension .s recognized as assembly language code for the IA-32.

Intel Compiler Features

Here we present the characteristics of Intel compilers as stated by the developer in the user manual with some of our comments.

• Significant optimization
  Apparently, this means optimizing the code at a high level, i.e. first of all, various loop transformations, which almost all compilers do with greater or less success
• Floating point optimization
  Apparently, this means, first of all, the maximum use of commands implemented at the hardware level
• Interprocedural Optimizations
  those. global optimization of the entire program, as opposed to ordinary optimization, which affects only the code of specific functions
• Profile-based optimization
  those. the ability to run a program in test mode, collect data on the time it took to pass certain code fragments inside frequently used functions, and then use this data for optimization
• Support for the SSE instruction set in Pentium III processors
  note: for computing tasks, SSE2 commands are of more interest, i.e. vector commands over 64-bit real numbers, but they are only supported on Pentium 4 processors, which we don’t have at our disposal yet
• Automatic vectorization
  those. again, using SSE and SSE2 commands inserted automatically by the compiler
• OpenMP support for programming on SMP systems
  note: on a cluster it is recommended to primarily use the MPI interface; widespread use of OpenMP on the cluster is not expected and such experiments have not yet been carried out; but it probably makes sense to use libraries (BLAS, etc.) that are parallelized for shared memory.
• Data Prefetching
  those. Apparently, the use of preload commands from memory into the data cache, which will be needed after some time
• "Dispatching" code for different processors
  those. the ability to generate code for different processors in a single executable file, which allows you to take advantage of the latest processors to achieve the highest performance on them, while maintaining binary compatibility of programs with earlier processors; On our cluster this is not relevant yet, because only Pentium III processors are used, and programs compiled on the cluster are not supposed to be transferred and run on other machines

Basic compiler options

The most interesting, of course, are the code optimization options. Most of the options are common to the C++ and Fortran compilers. More detailed description options in English user manuals.

Performance

According to the results of running the SPEC CPU2000 tests, published on the ixbt.com server, Intel compilers version 6.0 were almost universally better compared to gcc compilers versions 2.95.3, 2.96 and 3.1, and PGI version 4.0.2. These tests were conducted in 2002 on a computer with a Pentium 4/1.7 GHz processor and RedHat Linux 7.3.

According to tests conducted by Polyhedron, the Intel Fortran compiler version 7.0 was almost universally superior to other Fortran 77 compilers for Linux (Absoft, GNU, Lahey, NAG, NAS, PGI). Only in some tests the Intel compiler is slightly inferior to the Absoft, NAG and Lahey compilers. These tests were performed on a computer with a Pentium 4/1.8 GHz processor and Mandrake Linux 8.1.

Intel compilers version 9.1 also outperform gcc compilers, and show performance comparable to Absoft, PathScale and PGI.

We will be grateful to those users and readers who send us data on the impact of the choice of compiler (GCC or Intel) and optimization options on the speed of work on their real-life problems.

Libraries

The C language compiler uses a runtime library developed within the GNU project ( libc.a).

The following libraries are supplied with the Intel C++ compiler:

• libcprts.a- runtime library of the C++ language developed by Dinkumware.
• libcxa.a- additional runtime library for C++ development by Intel.
• libimf.a- a library of mathematical functions developed by Intel, which includes optimized and high-precision implementations of trigonometric, hyperbolic, exponential, special, complex and other functions (for more details, see the list of functions).
• libirc.a- runtime support for profiling (PGO) and code dispatch depending on the processor (see above).
• libguide.a- OpenMP implementation.

This list contains static libraries, but for most of them there are also dynamic ones, i.e. options connected during startup ( .so).

The following libraries are supplied with the Fortran compiler: libCEPCF90.a, libIEPCF90.a, libintrins.a, libF90.a, the library of mathematical functions libimf.a is also used.

Building the executable file

Libraries can be connected statically (during build) or dynamically (during program startup). The dynamic approach allows you to reduce the size of the executable file and allows you to share the same copy of the library in memory, but for this you need to install a complete set of dynamic libraries used on each node where the programs will be run.

Thus, if you installed the Intel compiler on your Linux machine and want to run the compiled executable files on other machines, then you need to either use a static build (which is easier) or copy the Intel dynamic libraries to these machines (usually from a directory like /opt/intel /compiler70/ia32/lib) to one of the directories listed in the /etc/ld.so.conf file, and also make sure that the same set of GNU/Linux dynamic libraries are installed on these machines.

By default, all Intel development libraries (except libcxa.so) are linked statically, and all Linux system libraries and GNU libraries are linked dynamically. Using the option -static you can force the collector (link editor) to connect all libraries statically (which will increase the volume executable file), and using the option -i_dynamic You can dynamically link all Intel development libraries.

When connecting additional libraries using the view option -llibrary you may need to use the option -Ldirectory to specify the path where the libraries are located.

Using options -Bstatic And -Bdynamic you can explicitly specify dynamic or static connection of each of the libraries specified in command line.

Using the option -c assembly of the executable file is disabled and only compilation is performed (object module generation).

Sharing modules in Fortran and C

To share modules written in Fortran and C, you need to agree on the naming of procedures in object modules, the passing of parameters, and access to global variables, if any.

By default, the Intel Fortran compiler converts procedure names to lowercase and adds an underscore to the end of the name. The C compiler never changes function names. Thus, if we want to call a function or procedure FNNAME implemented in C from a Fortran module, then in the C module it should be called fnname_.

The Fortran compiler supports the option -nus [filename], which allows you to disable the addition of underscores to internal procedure names. If a file name is specified, this is done only for procedure names listed in the specified file.

By default, in Fortran parameters are passed by reference, and in C they are always passed by value. Thus, when calling a Fortran procedure from a C module, we must pass pointers to the corresponding variables containing the values ​​of the actual parameters as parameters. When writing a function in C that will need to be called from a Fortran module, we must describe the formal parameters as pointers to the corresponding types.

In C modules, it is possible to use COMMON blocks defined inside Fortran modules (for more information, see Intel Fortran Compiler User's Guide, chapter Mixing C and Fortran).

Sharing Intel and GCC compilers

C object modules produced by the Intel C++ compiler are compatible with modules produced by the GCC compiler and the GNU C library. Thus, these modules can be used together in a single program compiled using the icc or gcc commands, but it is recommended to use icc to correctly include Intel libraries.

The Intel compiler supports a number of non-standard C language extensions used by the GNU project and supported by the GCC compiler (but not all of them, see here for more details).

The user manual does not say anything about the compatibility of object modules in the C++ and Fortran languages; apparently, it is not supported.

Standards support

Intel C++ Compiler 7.0 for Linux supports the ANSI/ISO C language standard (ISO/IEC 9899/1990). It is possible to establish strict compatibility with the ANSI C standard ( -ansi) or extended ANSI C dialect ( -Xa). When using the option -c99

In the previous issue of the magazine, we discussed products from the Intel VTune Performance Analyzer family - performance analysis tools that are deservedly popular among application developers and allow detection in code team applications, which waste too much CPU resources, giving developers the opportunity to identify and eliminate potential narrow places, associated with similar sections of code, thereby speeding up the application development process. Note, however, that the performance of applications largely depends on how efficient the compilers used in their development are, and what features hardware they are used when generating machine code.

The latest versions of the Intel Intel C++ and Intel Fortran compilers for Windows and Linux allow you to gain application performance benefits for systems based on Intel processors Itanium 2, Intel Xeon and Intel Pentium 4 up to 40% compared to existing compilers from other manufacturers due to the use of such features of these processors as Hyper-Threading technology.

Differences associated with code optimization by this family of compilers include the use of a stack for performing floating-point operations, interprocedural optimization (IPO), optimization in accordance with the application profile (Profile Guided Optimization (PGO), preloading data into the cache (Data prefetching), which avoids latency associated with memory access, support for the characteristic features of Intel processors (for example, extensions for streaming data processing Intel Streaming SIMD Extensions 2, characteristic of the Intel Pentium 4), automatic parallelization of code execution, application creation, running on several different types processors when optimizing for one of them, tools for “predicting” subsequent code (branch prediction), expanded support for working with execution threads.

Note that Intel compilers are used in such well-known companies as Alias/Wavefront, Oracle, Fujitsu Siemens, ABAQUS, Silicon Graphics, IBM. According to independent testing conducted by a number of companies, the performance of Intel compilers is significantly higher than the performance of compilers from other manufacturers (see, for example, http://intel.com/software/products/compilers/techtopics/compiler_gnu_perf.pdf).

Below we will look at some features latest versions Intel compilers for desktop and server operating systems.

Compilers for the Microsoft Windows platform

Intel C++ Compiler 7.1 for Windows

Intel C++ Compiler 7.1 is a compiler released earlier this year that provides highly optimized code for the Intel Itanium, Intel Itanium 2, Intel Pentium 4 and Intel Xeon processors, as well as the Intel Pentium M processor using Intel Centrino technology and intended for use in mobile devices.

The specified compiler is fully compatible with the development tools Microsoft Visual C++ 6.0 and Microsoft Visual Studio .NET: it can be built into the corresponding development environments.

This compiler supports ANSI and ISO C/C++ standards.

Intel Fortran Compiler 7.1 for Windows

Intel Fortran Compiler 7.1 for Windows, also released earlier this year, allows you to create optimized code for Intel Itanium, Intel Itanium 2, Intel Pentium 4 and Intel Xeon, Intel Pentium M processors.

This compiler is fully compatible with Microsoft Visual C++ 6.0 and Microsoft Visual Studio .NET development tools, that is, it can be built into the corresponding development environments. In addition, this compiler allows you to develop 64-bit applications for operating systems running on Itanium/Itanium 2 processors using Microsoft Visual Studio on a 32-bit Pentium processor using the 64-bit Intel Fortran Compiler. When debugging code, this compiler allows you to use a debugger to Microsoft platforms.NET.

If you have the Compaq product installed, Visual Fortran 6.6 can be used instead of the original Intel Fortran Compiler 7.1, since these compilers are compatible at the level source code.

Intel Fortran Compiler 7.1 for Windows is fully compatible with the ISO Fortran 95 standard and supports the creation and debugging of applications containing code in two languages: C and Fortran.

Compilers for the Linux platform

Intel C++ Compiler 7.1 for Linux

Another compiler that was released at the beginning of the year, Intel C++ Compiler 7.1 for Linux, allows you to achieve a high degree of code optimization for Intel Itanium, Intel Itanium 2, Intel Pentium 4, Intel Pentium M processors. This compiler is fully compatible with the GNU C compiler at the source code and object modules, which allows you to migrate applications created using GNU C to it without additional costs. The Intel C++ Compiler supports the C++ ABI (an addition to the Linux kernel that allows you to run under Linux control compiled code for other platforms such as early SCO operating systems, early versions Sun Solaris, etc.), which means full compatibility with the gcc 3.2 compiler at the binary code level. Finally, with Intel C++ Compiler 7.1 for Linux, you can even recompile the Linux kernel by making a few minor changes to its source code.

Intel Fortran Compiler 7.1 for Linux

The Intel Fortran Compiler 7.1 for Linux allows you to create optimized code for Intel Itanium, Intel Itanium 2, Intel Pentium 4, Intel Pentium M processors. This compiler is fully compatible with the Compaq Visual Fortran 6.6 compiler at the source code level, allowing you to recompile applications using it created using Compaq Visual Fortran, thereby increasing their performance.

In addition, the specified compiler is compatible with such utilities used by developers as the emacs editor, the gdb debugger, and the make application build utility.

Like the Windows version of this compiler, Intel Fortran Compiler 7.1 for Linux is fully compatible with the ISO Fortran 95 standard and supports the creation and debugging of applications containing code in two languages: C and Fortran.

It should be especially emphasized that a significant contribution to the creation of the listed Intel compilers was made by specialists from the Intel Russian Software Development Center in Nizhny Novgorod. More detailed information Information about Intel compilers can be found on the Intel Web site at www.intel.com/software/products/.

The second part of this article will be devoted to Intel compilers that create applications for mobile devices.

Examples of real hacks: Intel C++ 7.0 Compiler - Archive WASM.RU

...the Intel C++ 7.0 compiler downloaded late at night, at about five in the morning. I really wanted to sleep, but I was also torn by curiosity: whether the protection had been strengthened or not. Deciding that until I figured out the protection I still wouldn’t sleep, I opened new console, and resetting the system variables TEMP and TMP to the C:\TEMP directory, hastily typed indecently long name installer W_CC_P_7.0.073.exe on the command line (the need to set the TEMP and TMP variables is explained by the fact that in Windows 2000 they by default point to a very deeply nested directory, and the Intel C++ installer - and not only it - does not support paths of such a huge size).

It immediately became clear that the protection policy had been radically revised and now the presence of a license was checked already at the stage of installing the program (in version 5.x the installation was carried out without problems). OK, we give the dir command and look at the contents of what we now have to fight with:

Contents of the folder C:\TMP\IntelC++Compiler70

17.03.2003 05:10

html

17.03.2003 05:11

x86

17.03.2003 05:11

Itanium

17.03.2003 05:11

notes

06/05/2002 10:35 45 056 AutoRun.exe

07/10/2001 12:56 27 autorun.inf

10/29/2002 11:25 2 831 ccompindex.htm

10/24/2002 08:12 126,976 ChkLic.dll

10/18/2002 22:37 552 960 chklic.exe

17.10.2002 16:29 28 663 CLicense.rtf

17.10.2002 16:35 386 credist.txt

16.10.2002 17:02 34 136 Crellnotes.htm

03/19/2002 14:28 4 635 PLSuite.htm

02/21/2002 12:39 2 478 register.htm

02.10.2002 14:51 40 960 Setup.exe

02.10.2002 10:40 151 Setup.ini

07/10/2001 12:56 184 setup.mwg

19 files 2,519,238 bytes

6 folders 886,571,008 bytes free

Yeah! The setup.exe installation program takes only forty-odd kilobytes. Very good! It is unlikely that you can hide serious protection in such a volume, and even if so, this tiny file is worth nothing to analyze in its entirety - down to the last byte of the disassembler listing. However, it is not a fact that security code located exactly in setup.exe, it can be located in another place, for example... ChkLic.dll/ChkLic.exe, which together occupy a little less than seven hundred kilobytes. Wait, what is ChkLic? Is this short for Check License or what?! Hmm, the guys from Intel obviously have serious problems with a sense of humor. It would be better if they called this file "Hack Me" honestly! Okay, judging by the volume, ChkLic is the same FLEX lm, and we have already encountered it (see "Intel C++ 5.0 Compiler") and have a rough idea of ​​how to break it.

We give the command “dumpbin /EXPORTS ChkLic.dll” to examine the exported functions and... hold on tightly to Klava so as not to fall off the chair:

Dump of file ChkLic.dll

1. Section contains the following exports for ChkLic.dll

   0 characteristics

   3DB438B4 time date stamp Mon Oct 21 21:26:12 2002

3. 1 number of functions

   1 number of names

   ordinal hint RVA name

   1 0 000010A0 _CheckValidLicense

Damn it! The protection exports just one single function with the wonderful name CheckValidLicense. “Wonderful” - because the purpose of the function becomes clear from its name and it becomes possible to avoid painstaking analysis of disassembler code. Well, they've lost all interest... it would be better if they exported it ordinally or something, or at least christened it with some kind of intimidating name like DES Decrypt.

...were daydreaming! Okay, let's get back to our sheep. Let's think logically: if all the security code is concentrated directly in ChkLic.dll (and, judging by the “hinged” nature of the protection, this is indeed the case), then all “protection” comes down to calling CheckValidLicense from Setup.exe and checking the result returned by it. Therefore, to “hack” it is enough just to lose ChkLic.dll, forcing the ChekValidLicense function to always return... and, by the way, what should it return? More precisely: what exactly is the return value that corresponds to a successful license check? No, don't rush to disassemble setup.exe to determine it, because possible options not so much anymore: either FALSE or TRUE. Are you betting on TRUE? Well, in a sense, this is logical, but on the other hand: why did we actually decide that the CheckValidLicense function returns exactly the flag of the success of the operation, and not the error code? After all, it must somehow motivate the reasons for refusing to install the compiler: the file with the license was not found, the file is damaged, the license has expired, and so on? Okay, let's try to return zero, and if that doesn't work, we'll return one.

OK, buckle up, let's go! We launch HIEW, open the file ChkLic.dll (if it doesn’t open, remember the gophers three times, temporarily copy it to the root or any other directory that does not contain special characters in its name that hiew doesn’t like). Then, turning again to the export table obtained using dumpbin, we determine the address of the CheckValidLicense function (in this case 010A0h) and through “10A0” we go to its beginning. Now, we cut it “live”, overwriting the old code “XOR EAX, EAX/RETN”. 4". Why exactly "REN 4" and not just "RET"? Yes, because the function supports the stdcall convention, which you can find out by looking at its epilogue in HIEW"e (just scroll down the disassembler screen until meet RET).

Let's check... It works!!! Despite the lack of a license, the installer begins the installation without asking any questions! Therefore, the defense fell. Oh, we can’t believe that everything is so simple and, in order not to sit stupidly staring at the monitor waiting for the program installation process to complete, we use our favorite IDA disassembler on setup.exe. The first thing that catches your eye is the absence of CheckValidLicense in the list of imported functions. Maybe it somehow launches the ChkLic.exe file? We try to find the corresponding link among the automatically recognized lines: “~View aNames”, “ChkLic”... yeah, the line “Chklic.exe” is not there at all, but “Chklic.dll” is detected. Yeah, I see, that means the ChkLic library is loaded by explicit linking via LoadLibrary. And following the cross-reference confirms this:

Text:0040175D push offset aChklic_dll ; lpLibFileName

Text:00401762 call ds:LoadLibraryA

Text:00401762 ; load ChkLic.dll ^^^^^^^^^^^^^^^^^

Text:00401762 ;

Text:00401768 mov esi, eax

Text:0040176A push offset a_checkvalidlic ; lpProcName

Text:0040176F push esi ; hModule

Text:00401770 call ds:GetProcAddress

Text:00401770 ; get the address of the CheckValidLicense function

Text:00401770 ;

Text:00401776 cmp esi, ebx

Text:00401778 jz loc_40192E

Text:00401778 ; if there is no such library, then exit the installation program

Text:00401778 ;

Text:0040177E cmp eax, ebx

Text:00401780 jz loc_40192E

Text:00401780 ; if there is no such function in the library, then exit the installation

Text:00401780 ;

Text:00401786 push ebx

Text:00401787 call eax

Text:00401787 ; call the ChekValidLicense function

Text:00401787 ;

Text:00401789 test eax, eax

Text:0040178B jnz loc_4019A3

Text:0040178 ; if the function returned non-zero, then exit the installation program

Incredibly, this terribly primitive defense is built exactly like this! Moreover, the half-meter file ChkLic.exe is not needed at all! And why was it worth dragging it from the Internet? By the way, if you decide to save the compiler distribution (attention: I didn’t say “distribute”!), then to save disk space, ChkLic.* can be erased: either by deleting setup.exe, forever weaning it from accessing them, or by simply creating your own ChkLic.dll, exporting the stdcall function CheckValidLicence of the form: int CheckValidLicence(int some_flag) ( return 0;)

Well, while we were discussing all this, the installer finished installing the compiler and successfully completed its work. Is it interesting whether the compiler will start or is all the fun just beginning? We feverishly go down the branched hierarchy of subfolders, find icl.exe, which, as one would expect, is located in the bin directory, click and... The compiler naturally does not start, citing the fact that "icl: error: could not checkout FLEX lm license" , without which he cannot continue his work.

It turns out that Intel used multi-level protection and the first level turned out to be a crude protection against fools. Well! We accept this challenge and, based on our previous experience, automatically look for the LMGR*.DLL file in the compiler directory. Useless! This time there is no such file here, but it turns out that icl.exe has gained a lot of weight, exceeding the six hundred kilobyte mark... Stop! Didn't the compiler developers link this same FLEX lm with static linking? Let's see: in Intel C++ 5.0, the sum of the sizes of lmgr327.dll and icl.exe was 598 KB, and now icl.exe alone occupies 684 KB. Taking into account the adjustment for natural senile “obesity”, the figures agree very well. So, after all, FLEX lm! Oh oh! But now, without symbolic function names, it will be much more difficult to break the protection... However, let’s not panic ahead of time! Let's think, just calmly! It is unlikely that the development team completely rewrote all the code that interacts with this “envelope” protection. Most likely, its “improvement” ended with just a change in the type of layout. And if so, then the chances of hacking the program are still great!

Remembering that last time the security code was in the main function, we, having determined its address, simply set a breakpoint and, waiting for the debugger to pop up, stupidly trace the code, alternately looking at the debugger, then at the program output window: has the Is there an expletive message? At the same time, we mark all the conditional transitions we encounter on a separate piece of paper (or put it in our own memory, if you want), not forgetting to indicate whether each conditional transition was executed or not... Stop! You and I were chatting about something, but an abusive message has already popped up! OK well! Let's see what conditional transition corresponded to it. Our records show that the last jump encountered was the JNZ conditional jump, located at address 0401075h and "reacting" to the result returned by sub_404C0E:

Text:0040107F loc_40107F: ; CODE XREF: _main+75^j

Text:0040107F mov eax, offset aFfrps ; "FFrps"

Text:00401084 mov edx, 21h

Text:00401089 call sub_404C0E

Text:0040108E test eax, eax

Text:00401090 jnz short loc_40109A

Obviously, sub_404C0E is the very protective procedure that checks the license for its presence. How to trick her? Well, there are many options... Firstly, you can thoughtfully and scrupulously analyze the contents of sub_404C0E to find out what exactly it checks and how exactly it checks. Secondly, you can simply replace JNZ short loc_40107F with JZ short loc_40107F or even NOP, NOP. Thirdly, the return result check command TEST EAX, EAX can be turned into a zero command: XOR EAX, EAX. Fourthly, you can make sub_404C0E itself disappear so that it always returns zero. I don’t know about you, but I liked method number three the most. We change two bytes and launch the compiler. If there are no other checks of its “licensing” in the protection, then the program will work and, accordingly, vice versa. (As we remember, in the fifth version there were two such checks). Amazingly, the compiler no longer complains and works!!! Indeed, as one would expect, its developers did not strengthen the protection at all, but, on the contrary, even weakened it! Chris Kaspersky

You are not a slave!
Closed educational course for children of the elite: "The true arrangement of the world."
http://noslave.org

Material from Wikipedia - the free encyclopedia