Add support for compilation on Windows using mingw32 #287
Conversation
|
thanks Paul. A quick rebase? There seem to be merge conflicts. |
|
Cool :-) |
|
@soumith, yes, I saw the conflict, but wasn't sure what to do about it as it's caused by another merge you just did: I can probably turn it into this without much harm: |
pkulchenko
force-pushed
the
torch-compile-mingw32
branch
from
5b8dd40
to
6f9eaf9
Compare
|
@soumith, resolved, force-pushed. |
Add support for compilation on Windows using mingw32
|
Thanks paul :) |
|
I haven't seen any problems yet, but will let you know if anything comes up ;). |
|
Hi @pkulchenko thanks for your work! Just wondering how about the installation of other torch packages? such as the nngraph, nn, optim etc packages using mingw for installation on Windows 8? |
|
I guess I'll need to do the same for those packages as well. I'll take a look in few days and will link to this ticket to track the progress... I'll probably start with nn as it's by far the most popular of the packages; @soumith, anything else you may suggest related to this? |
|
I think this effort can largely base itself on the work of @diz-vara 's efforts. He already patched most of the core packages for Windows support using VC++. We will actively not block this, but we dont have the resources (or motivation) to actively support Windows either. |
|
@diz-vara indeed did really great work for torch7 installation on Windows! |
|
Main packages (torch, nn, nnx, optim, luafilsystem, graph, nngraph, paths, xlua) just work in my VS2013 (x64) configuration from official repository. |
|
@soumith, it's a reasonable position, but I think it's a short step to add mingw support as well (since VC++ is already supported).
@diz-vara, it should mostly work out of the box with a bit of configuration as it requires 64bit luasocket. I recently compiled it for the case like this and it's available here. ZeroBrane Studio allows you to debug 64bit apps even though it's running as 32bit app itself. You can ping me in IRC or over email if you run into any issues. |
|
@hohoCode, I successfully compiled @soumith, @diz-vara, any idea why I got a linker error on missing Lua symbols? I fixed it by adding lua51.dll to the list of the libraries, but I'm wondering how it works with VC++. The following patch fixes it for me: diff --git a/CMakeLists.txt b/CMakeLists.txt
index 1bc2bad..3ffd2a1 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -49,5 +49,9 @@ ADD_TORCH_PACKAGE(nn "${src}" "${luasrc}")
TARGET_LINK_LIBRARIES(nn luaT TH)
+IF(LUALIB)
+ TARGET_LINK_LIBRARIES(nn ${LUALIB})
+ENDIF()
+
INSTALL(DIRECTORY "doc" DESTINATION "${Torch_INSTALL_LUA_PATH_SUBDIR}/nn")
INSTALL(FILES "README.md" DESTINATION "${Torch_INSTALL_LUA_PATH_SUBDIR}/nn")I then add |
|
Paul @pkulchenko, I don't think we may bother @soumith (and other Torch/FAIR people) with Windows questions. There may be a lot of problems concerning libraries, build process, CMake and nmake behavior etc - and those are not Torch problems. #if !defined LUA_LIB & !defined LUA_CORE & !defined luajit_c & !defined _LJ_ARCH_H & !defined _LJ_DEF_H & !defined _LJ_OBJ_H pragma comment( lib, "libluajit.lib")endif |
|
@diz-vara, it's an option, but I prefer not to modify the source files unless absolutely necessary. At least this explains how it works for you without that library being directly referenced. I'll submit a pull request after I finish my tests as it seems to be working fine otherwise. |
|
There's a question on reddit about running torch on windows. Is there any working way of running torch on windows currently that we can point him to? https://www.reddit.com/r/MachineLearning/comments/3dgvvu/how_to_make_rnn_learning_faster/ He wants to run RNNs. |
|
See the last answer on this thread: https://groups.google.com/forum/#!searchin/torch7/windows/torch7/A5XUU4u9Tjw/yvSJCBuLj4oJ That is the most complete one as far as I know. But that instructions are based on an older fork of torch that was frozen. |
|
I've build cutorch on Windows (using my version of luajit-rocks - https://github.com/diz-vara/luajit-rocks), and it works. But it was done manually (with CMake-GUI and VS2013). I'll try to automate it - and pull the updates. |
Cool. Sounds good. I'm currently partial to Paul Graham's idea that it's better to have a couple of people who really really want to use your project, than a zillion who kind of might do maybe |
|
That was my question on reddit; I followed the link to the installer and thought it was exactly what I needed. Unfortunately it's missing the lfs and nngraph packages that the RNN script I'm using uses, and I can't get them to install using the torch-rocks.exe included in the installer... something about not finding the manifest, even when I point it at https://github.com/torch/rocks using --source. I had a luafilesystem.rockspec file that I tried running make on (again with torch-rocks.exe) but it complains about chmod not being a command and not being able to open source file lfs.c. |
|
@PAK90 NB! lfs rock is called 'luafilesystem' - so you need to 'luarocks install luafilesystem' It works also with qtlua, image, cutroch and cunn - but it requires some (a lot of) 'manual' work' |
|
@soumith I downloaded and installed that Wintorch file. I've almost got it running with my RNN script, but it's missing some luarocks packages. I try to use torch-rocks.exe to install the packages but it always comes back with error messages regarding loading the manifest: I know that the manifest file works fine, and I know the package I need is there. Below that error message, I also get: Not sure if they're related. The second one is daft as well; I have the MinGW path in my PATH variable, and I copied the chmod.exe file into the Torch7/bin folder itself (same one as where torch-rocks.exe is), yet it still complains. Failing that, is there a way to point WinTorch at my luarocks install folder? I can easily use that luarocks install to download the rocks packages I need, but I can't copy them across to the Torch7 folder since all they are is a .rockspec file. |
|
@PAK90 : you can try to add directory with 'chmod' (and other unix-like staff) to your path. |
|
in python scripts, I notice sometimes that I have to give the Windows path, rather than the mingw path, when I execute external commands, so could be worth trying eg both |
|
Hi @vladimirdlc , it requires trivial change in torch to get the whole torch stack compiled for x86. I have built x86 torch locally, and have updated distro-win to support x86. Please pull latest changes and retry. According to your log, are you using x64 conda? x86 conda for x86 torch, x64 conda for x64 torch. In case the build is messed up between x86 and x64, try uninstall torch before a fresh new build of torch. install\torch-activate.cmd
uninstall.bat
install.bat |
|
Hi @BTNC, it's working! :D |
|
Hi @vladimirdlc , the readme contains instructions on how to install dpnn as a example of luarocks install not working for some torch packages. Actually it will work when dpnn's rockspec in torch rock server is up-to-date. Some torch packages can not be simply luarocks installed on windows. |
|
That explains it, thanks again. |
|
By the way, I see you are using x64 arm command prompt, are you trying to cross compile arm torch from x64 toolchain? I did not test cross compile torch with msvc. |
|
I should have ran the wrong prompt. |
|
@BTNC
Then:
|
|
@TyrionLanister ,according to the following log:
It looks like x64 trepl is trying to link with x86 editline static library which will not work. The reason I could guess is either trepl is using the downloaded prebuild wineditline static library which machine type is x86 (this is hardly possible if install.bat is called from a fresh command prompt and wineditline is rebuilt locally), or a x86 installation was run before this x64 installation. The x64 run will not call install-deps.bat again if last x86 run of install-deps.bat succeeded which leaves wineditline compiled for x86. In this case I suggest you to uninstall it and reinstall from a clean state: If my guess is not your case, please detail your environment and procedure. |
|
Thanks it worked once I used "VS2015x64 native tools command prompt" . |
|
@BTNC I have been able to install CUDA 8.0 in my windows 10 with NVIDIA GT740 M but I get this error *Unknown CUDA Architecture Name " nvcc " in CUDA_SELECT_NVCC_ARCH_FLAGS once installing cutorch.
|
|
@TyrionLanister , you git cloned old cutorch before this commit. cutorch can only be compiled with msvc since this commit. I suggest you stick with distro way of installation for msvc, since distro refers to the snapshot of torch that can be fully compiled with msvc2015. To install cutorch (and cunn etc) manually, I suggest you use the code synced by distro, and follow the commands in install.bat |
|
@bntc. Look at the error where it shows the build process using internally installed "C:\Program Files\CMake\share\cmake-3.7\Modules\FindCUDA\select_compute_arch.cmake" file so I updated my internal cmake file with your latest commit of "select_compute_arch.cmake" |
|
@TyrionLanister , it turns out cmake has introduced the select_compute_arch.cmake file since 3.7 and cutorch is not using its own select_compute_arch.cmake. Thanks for the report. |
|
On windows7 x64 msvc 10 I got this : +++++++ Installing common lua packages lfs.obj : warning LNK4197: export 'luaopen_lfs' specified multiple times; using first specification Setting environment for using Microsoft Visual Studio 2010 x64 tools. fatal: unrecognized input lua_cjson.obj : warning LNK4197: export 'luaopen_cjson' specified multiple times; using first specification Error: Build error: Failed compiling module cjson.dll |
|
Hi @etaoinbe , lua-cjson is compiled with the help of this patch and is installed by this line. It is only tested for vs2015. I do not have vs2010 in hand. Please feel free to update that patch to work with vs2010. Please temporarily try BTNC/distro-win in case you were using torch/distro, because latest workable version has not been committed to torch/distro due to one cutorch test error. |
|
i am already using btnc/distro-win the patch does get applied but linker does not find isnan and isinf looks like i ran into http://stackoverflow.com/questions/38441740/where-is-isnan-in-msvc-2010 will have another try at vs2015 logfile: batchfile that gave above result: cd \data\git |
|
with vs2015 I get failing tests cmTC_c76c6.exe - System ErrorThe program can't start because ucrtbased.dll is missing from your computer. Try reinstalling the program to fix this problem.OKfollowing is supposed to fix that but it doesnt https://qualapps.blogspot.be/2016/05/remote-debugging-visual-c-2015.html Warning: unmatched variable INTEL_MKL_DIR -- Try OpenMP C flag = [/openmp] blabla... c:\data\git\distro-win\pkg\torch\lib\th\generic/THTensorConv.c(1000): warning C4244: '=': conversion from 'double' to 'long', possib Error: Build error: Failed building. |
|
@etaoinbe , you can refer to this patch about how lua-cjson is patched for vs2015, and update it accordingly for vs2010. According to your logt7.txt file, it seems winedit can not be compiled by vs2010. It is used for trepl package and is not a blocker to use torch. Regarding missing ucrtbased.dll on win7, I see some references after searched google and here is a link shows some people solved it in different ways. Regarding the error of THAtomic, I am not using win7 and I am not sure why vs2015 on win7 can have that error. You can check your log (the part you did not paste in your comment) whether it contains Atomics: using MSVC intrinsics or Atomics: using pthread. Furthermore you can update this part to work in your environment with msvc intrinsics. |
|
Apparently it is necessary to install full visual studio, command line tools are not enough. build appears to have succeeded apart from qt stuff. Unfortunately I can not turn off the virusscanner on this laptop. If I would copy over this buildresult to another machine would you expect it to work as well ? |
|
Since all the installation information are stored with full path, it may work if it is copied to the same directory in the other machine with same conda environment setup, however I did not try that. |
|
the qt stuff compiled after a few more tries, now trying to run a demo luarocks install nnx |
|
so, is there a way to use cutorch in jkjung-avt's windows build? |
|
I just rebuilt Torch on Windows in both 32-bit and 64-bit configurations and made the resulting binaries available here: https://github.com/hiili/WindowsTorch To use them, simply download to C:\torch, then run setpaths.cmd and luajit.exe. These should be fully self-contained builds (tested on two relatively cleanish machines), but I haven’t had an opportunity to test them on a fully clean machine yet. If anyone tries them, then please let us know whether they work. Thanks! I recorded all commands issued during building into the git logs. ( ping @hughperkins ) |
|
@hiili |
|
@stillcold , you need to |
|
It is works for me! Thanks very much! |
|
Could this Windows version of Torch run this code? |
|
Hello, I downloaded Windows Torch 64-bit version, and manually added paths like shows following image (as setpaths didn't seem to work):
After doing cd C:\torch\bin\luajit, require('torch') and torch.test(), it seems to work, as the word PASS is printed, though the encoding doesn't shows correctly:
But when I try to run it on Visual Studio Code, it doesn't work
Any ideas about it? |
|
@ayrtondenner Answered here: hiili/WindowsTorch#4 |
This patch enables Torch7 compilation on Windows using mingw32 (or tdm-gcc with msys/msys2).
The steps needed:
torch/cwrapandtorch/pathsRelated ticket: #150. I've tested this under Windows 8.1 using msys2 and tdm-gcc-4.8, but it should work with other versions and configurations of mingw.