[WIP: June2023] Deep Q-Learning using TorchSharp #710
Replies: 3 comments 6 replies
-
using Torch;
using System;
namespace DeepQLearning
{
class Program
{
static void Main(string[] args)
{
// Define the neural network
var model = new Sequential();
model.Add(new Linear(4, 128));
model.Add(new ReLU());
model.Add(new Linear(128, 2));
// Define the optimizer
var optimizer = new Adam(model.Parameters(), 0.001);
// Define the loss function
var loss = new MSELoss();
// Define the environment
var env = Gym.Make("CartPole-v0");
// Train the model
int episodeCount = 1000;
int stepsPerEpisode = 200;
int maxSteps = episodeCount * stepsPerEpisode;
int stepCount = 0;
int episode = 0;
while (stepCount < maxSteps)
{
env.Reset();
for (int step = 0; step < stepsPerEpisode; step++)
{
// Get the current state
var state = env.Observation;
// Choose an action based on the current state
var tensor = new Tensor(state, new[] { 1, state.Length });
var qValues = model.Forward(tensor);
var action = qValues.Max().Item2;
// Take the action and observe the result
var result = env.Step(action);
var nextState = result.Observation;
var reward = result.Reward;
var done = result.Done;
// Calculate the target Q-value
var target = qValues.Clone();
if (done)
{
target[0, action] = reward;
}
else
{
var nextTensor = new Tensor(nextState, new[] { 1, nextState.Length });
var nextQValues = model.Forward(nextTensor);
var maxNextQ = nextQValues.Max().Item1;
target[0, action] = reward + 0.99 * maxNextQ;
}
// Calculate the loss and update the model
tensor.Reshape(new[] { 1, 4 });
var output = model.Forward(tensor);
optimizer.ZeroGrad();
var l = loss.Forward(output, target);
l.Backward();
optimizer.Step();
// Update the step count
stepCount++;
// Check if the episode is done
if (done)
{
break;
}
}
// Print the episode number
Console.WriteLine("Episode: " + episode);
episode++;
}
// Close the environment
env.Close();
}
}
}
|
Beta Was this translation helpful? Give feedback.
-
As far as TorchSharp goes, I wouldn't trust ChatGPT to get anything right. It hasn't seen enough code to train on, and so it's just making stuff up. |
Beta Was this translation helpful? Give feedback.
-
|
I think I reproduced solution from lecture in my repo hear. |
Beta Was this translation helpful? Give feedback.
-
|
This is cool. I still hope we can build out a gym in .NET and maybe some shareable components for Q-learning. I don't have the expertise or experience to do that, but it'd be very cool. |
Beta Was this translation helpful? Give feedback.
-
|
@NiklasGustafsson we now have at least two projects using TorchSharp on top of Gym.NET (partial implementation) |
Beta Was this translation helpful? Give feedback.
-
I'd like to help any way I can. I am building my own deep reinforcement learning lib with TorchSharp and using it for my own research, would be nice to integrate more closely. https://github.com/asieradzk/RL_Matrix I have distributed DRL, godot plugin and shiny DQN-Rainbow & PPO for 1d and 2d. Ease of swapping algos and options is key for me. |
Beta Was this translation helpful? Give feedback.
-
|
@asieradzk You are still new in your journey. The key of your training is to focus on the contributions await you. First, you need to trust that the recognition of your contribution must be earned. Change your project licensing to MIT, this will free yourself from the constraints that limit your potential. Second, you are in the right project and there is great need to what you are doing. Seize this opportunity and address the gap. No one is dared to complete the porting of Stable-baselines3 to c# using TorchSharp. Previous attempts have been tried, all failed. It is up to you to take up this challenge and if you manage to do it, present that achievement during your oral defense for your degree. |
Beta Was this translation helpful? Give feedback.
-
|
@GeorgeS2019 My lib is already (in some aspects) better than SB3. They dont have DQN-Rainbow or distributed DRL. So I am not porting SB3 to C#. Nobody needs another port of SB3, RLMatrix is geared towards use in game engines & asp.net core applications. I am writing my own code to the best of my ability in application of SOLID principles to reach feature parity (and eventually overcome) RLlib & Unity ML Agents (I am very close). I am also aiming towards reaching better performance (iterations/time) than RLib & SB3. If anything I am surprised that RLib, ML-Agents, SB3 need teams of researchers and years of work. I do equivalent of what they have accomplished ALONE with zero mentorship in much shorter time-frame. I think this just demonstrates superiority of C# in the development of deep reinforcement learning applications and algorithms. |
Beta Was this translation helpful? Give feedback.
-
|
The appropriate place to comment on the licensing choices of a particular repository is in that repository. Let's keep the discussion in this forum focused on the relevance (or irrelevance) of TorchSharp to the domain. |
Beta Was this translation helpful? Give feedback.
-
June 2023
#981 (comment)
mindmap root((Reinforcement<br/>Learning)) Definitions Interactions Environment Agent Elements State Action Strategy<br/>策略 Deterministic Policy<br/>确定性策略 Stochastic Policy<br/>随机性策略 State transfer probability<br/>状态转移概率 Rewards<br/>即时奖励 Others Episodes Trial Continuing Tasks Policy Policy based learning Value based learning Monte Carlo learning Temporal Difference Learning SARSA<br/>State Action Reward State Action QLearning Dynamic programming learning Policy iteration algorithm Policy Evaluation Policy Improvement Value iteration algorithm Markov Decision Process Markov Decision Process<br/>马尔科夫决策过程 Trajectory<br/>轨迹 Markov Process<br/>马尔科夫过程 Objective FunctionsFeb 2023
https://www.youtube.com/watch?v=217tCMsZu0I
Beta Was this translation helpful? Give feedback.
All reactions